User interface for remote control of medical devices

ABSTRACT

An interface for remotely controlling a medical device in a patient&#39;s body provides a two dimensional display of a three dimensional rendering of the operating region, and allows the user to select the orientation or location of the distal end of the medical device on the display and then operate a navigation system to cause the distal end of the medical device to approximately assume the selected orientation or location.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Application Ser. No.60/576,946, filed Jun. 4, 2004, incorporated herein by reference.

BACKGROUND OF THE INVENTION

This invention relates to the remote navigation of medical devices in apatient's body, and in particular to a user interface for controlling aremote navigation system.

Advances in technology have resulted in systems that allow a physicianor other medical professional to remotely control the orientation of thedistal of a medical device. It is now fairly routine steer the distalend of a medical device inside a patient's body by manipulating controlson the proximal end of the medical device. Recently magnetic navigationsystems have been developed that allow a physician to orient the distalend of a medical device using the field of an external source magnet.Other systems have been discussed for the automated remote orientationof the distal end of a medical device, for example by operatingmagnetostrictive or electrostrictive elements incorporated into themedical device. However the medical device is oriented, it is stilldifficult for a physician to visualize the procedure site (which is outof view inside the patient's body), to selected the desired direction inwhich to orient the distal end of the medical device and communicate theselected direction to the system in order to orient the distal end ofthe medical device in the selected direction.

SUMMARY OF THE INVENTION

The present invention relates to an interface to facilitate theselection of the desired direction in which to orient the distal end ofthe medical device and to communicate the selected direction to anavigation system in order to orient the distal end of the medicaldevice in the selected direction. While the present invention isdescribed primarily in connection with a magnetic navigation system, theinvention is not so limited, and can be used in connection with othernavigation systems, such as those that can orient the distal end of amedical device with mechanical means, electrostrictive elements,magnetostrictive elements, or otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an interface system according to theprinciples of this invention;

FIG. 2 is a schematic diagram of a possible implementation of theinterface for use in controlling a magnetic surgery system;

FIG. 3 is a schematic diagram of the display of a first preferredembodiment of the interface of this invention;

FIG. 4A is a view of the display of the first preferred embodiment ofthe interface of this invention, showing several points on the 3-Ddisplay pane and the desired orientation arrow;

FIG. 4B is a view of the display of the first preferred embodiment ofthe interface of the invention, showing several points on the 3-Ddisplay pane, a current direction vector and a desired direction vector;

FIG. 4C is a view of the display of the first preferred embodiment ofthe interface of the invention, showing the anatomical model in the 3-Ddisplay pane, with the picture-in-picture feature turned off;

FIG. 4D is a view of the display of the first preferred embodiment ofthe interface of the invention, showing the bull's eye display in the3-D display pane;

FIG. 4E is a view of the display of the first preferred embodiment ofthe interface of the invention, showing the bull's eye display in thepicture-in-picture portion of the 3-D display pane;

FIG. 4F is a view of the display of the first preferred embodiment ofthe interface of the invention, showing the bull's eye display in thepicture-in-picture portion of the 3-D display pane, and anatomical modelin the main 3-D display with the viewpoint changed from FIG. 4E;

FIG. 4G is a view of the display of the first preferred embodiment ofthe interface of the invention;

FIG. 4H is a view of the display of the first preferred embodiment ofthe interface of the invention;

FIG. 4I is a view of the display of the first preferred embodiment ofthe interface of the invention;

FIG. 4J is a view of the display of the first preferred embodiment ofthe interface of the invention;

FIG. 5 is an enlarged view of the 3-D display pane of the firstpreferred embodiment of the interface of this invention;

FIG. 6A and FIG. 6B are left anterior oblique (LAO) and right anterioroblique (RAO) images of the procedure site with desired orientationarrow and visualization surface superposed thereon;

FIG. 6C is an alternate implementation of the visualization surfacesuperposed thereon;

FIG. 7 is an enlarged view of the status pane of the first preferredembodiment of the interface of this invention;

FIG. 8A is an enlarged view of the 2-D navigation pane of the firstpreferred embodiment of the interface of this invention;

FIG. 8B is an enlarged view of an alternate embodiment of the 2-Dnavigation pane of FIG. 8A;

FIG. 9 is an enlarged view of the point navigation pane of the firstpreferred embodiment of the interface of this invention;

FIG. 10 is an enlarged view of the vector navigation pane of the firstpreferred embodiment of the interface of this invention;

FIG. 11 is an enlarged view of the bull's eye navigation pane of thefirst preferred embodiment of the interface of this invention;

FIG. 12 is a view of an x-ray image showing the projection of the bull'seye screen thereon;

FIG. 13 is an enlarged view of the menu bar of the first preferredembodiment of the interface of this invention;

FIG. 14 is a schematic diagram of the display of a second preferredembodiment of the interface of this invention;

FIG. 15 is a view of the display of the second preferred embodiment ofthe interface of this invention;

FIG. 16 is a view of the display of the second preferred embodiment ofthe interface of this invention, showing an alternate image in pane 308;

FIG. 17 is a view of the display of the second preferred embodiment ofthe interface of this invention, showing the use of target navigationpane 314;

FIG. 18 is a view of the display of the second preferred embodiment ofthe interface of this invention, showing a possible path of a medicaldevice;

FIG. 19 is a view of the display of the second preferred embodiment ofthe interface of this invention, showing an elliptical constellation ofpoints and a possible path of a medical device to the constellation;

FIG. 20 is a view of a display of a third preferred embodiment of aninterface in accordance with the principles of this invention, with twomain panes;

FIG. 21 is a view of a display of a third preferred embodiment of aninterface of the third embodiment, with four main panes;

FIG. 22A is a view of a display of the third preferred embodiment, withbiplanar images in the main panes, showing the selection of a point increating a predetermined branched path;

FIG. 22B is an enlarged view of the biplanar images in the main panes;

FIG. 23A is a view of a display of the third preferred embodiment, withbiplanar images in the main panes, showing the completion of apredetermined branched path;

FIG. 23B is an enlarged view of the biplanar images in the main pane ofFIG. 23A;

FIG. 24A is a view of a display of the third preferred embodiment, withbiplanar images in the main panes, showing the specification of anavigation field at an application point;

FIG. 24B is an enlarged view of the biplanar images in the main pane ofFIG. 24A;

FIG. 25A is a view of a display of the third preferred embodiment, withbiplanar images in the main panes, showing the specification of anavigation field at an application point;

FIG. 25B is an enlarged view of the biplanar images in the main pane ofFIG. 25A;

FIG. 26A is a view of a display of the third preferred embodiment, withbiplanar images in the main panes, showing the exemplary branched path;

FIG. 26B is an enlarged view of the biplanar images in the main pane ofFIG. 26A;

FIG. 27A is an enlarged view of a pattern navigation control pane of thethird preferred embodiment of this invention;

FIG. 27B is an enlarged view of the pattern navigation control paneafter navigation of the medical device to the first new position in thepattern;

FIG. 27C is an enlarged view of the pattern navigation control pane asthe medical device is moved from the first new position in the patternto the second new position;

FIG. 27D is an enlarged view of the pattern navigation control paneafter navigation of the medical device to the second new position in thepattern;

FIG. 28 is a view of a display of a third preferred embodiment of aninterface in accordance with the principles of this invention, showinganother control for specifying the direction of the magnetic field to beapplied;

FIG. 29 is a view of a display of a third preferred embodiment of aninterface in accordance with the principles of this invention, showinganother control for specifying the direction of the magnetic field to beapplied;

FIG. 30A is a view of a display of a third preferred embodiment of aninterface in accordance with the principles of this invention, showingan orientation element;

FIG. 30B is a view of a display of a third preferred embodiment of aninterface in accordance with the principles of this invention, showingan orientation element;

FIG. 30C is a view of a display of a third preferred embodiment of aninterface in accordance with the principles of this invention, showingan orientation element; and

FIG. 31 is a schematic view of the hardware for operating the userinterface used in a fourth preferred embodiment of this invention;

FIG. 32 is view of the main menu control screen of the fourth preferredembodiment of a user interface in accordance with the principles of thepresent invention;

FIG. 33 is a view of a display from the basic mode of the fourthpreferred embodiment of a user interface;

FIG. 34A is a view of a display from the EP mode of the fourth preferredembodiment of a user interface, in the right atrium mode;

FIG. 34B is a view of an alternative to the display of FIG. 34A;

FIG. 35 is a view of a closed spline indicator used in the sphericalobject navigation mode of the preferred embodiment of the presentinvention;

FIG. 36 is a view of an ostium indicator used in the spherical objectnavigation mode of the preferred embodiment of the present invention;

FIG. 37 is a view of a surface constellation indicator used in thespherical navigation mode of the preferred embodiment of the presentinvention

FIG. 38 is a view of a control pane for a catheter adjustment mode ofthe user interface of the preferred embodiment;

FIG. 39 is a view of a pane for a catheter adjustment mode of the userinterface of the preferred embodiment,

FIG. 40 is a view of the spherical navigation mode of the preferredembodiment of the user interface;

FIG. 41 is a view of the spherical navigation mode of the preferredembodiment;

FIG. 42 is an enlarged partial view of the spherical object used in thespherical object navigation mode of the preferred embodiment of thepresent invention;

FIG. 43A is a view of a display from the EP mode of the fourth preferredembodiment of a user interface, in the right atrium mode;

FIG. 43B is a view of an alternative to the display of FIG. 43A;

FIG. 44 is a view of a display from the EP mode of the fourth preferredembodiment of a user interface, in the full heart mode;

FIG. 45A a view of a display from the EP mode of the fourth preferredembodiment of a user interface, in the left atrium mode;

FIG. 45B is a view of an alternative to the display of FIG. 45A;

FIG. 46 is a view of a display from the CRT mode of the preferredembodiment of the user interface;

FIG. 47 is an alternate view of a display from the CRT mode of thepreferred embodiment of the user interface

FIG. 48A is a view of the display of an anatomical object navigationmode;

FIG. 48B is an alternate view of the selections made;

FIG. 49A is an enlarged view of Alignment window, shown in the automode; and

FIG. 49B is an enlarged view of the Alignment window, shown in themanual mode.

Corresponding reference numerals indicate corresponding parts throughoutthe several views of the drawings.

DETAILED DESCRIPTION OF THE INVENTION

This invention relates to an interface for a navigation system fororienting the distal end of a medical device inside a patient's body. Asshown in FIG. 1 the interface, indicated generally as 20, comprises aprocessor 22, a display 24, and an input device 26. The display 24preferably includes at least one monitor 28, which may be a flat panelLCD display which is small, compact, and less prone to interference. Theinput device 26 may include a keyboard 32, a mouse 34, a track ball 36,microphone 38, or other device for controlling a cursor on the display24.

A possible implementation of an interface system is indicated in FIG. 2,in which components of the interface are distrubuted in the procedureroom 50 where the patient is located, and a control room 52. The controlroom 52 is preferably adjacent the procedure room 50, and there may be awindow 54 between the control room and the procedure room to permitdirect observation of the patient, however the control room could beremote from the patient, and with the aid of the present interface, aphysician could conduct a procedure on a patient in the procedure from acontrol room on a different floor, in a different building, or even in adifferent city.

As shown in FIG. 2, a magnetic surgery suite comprising a patient bed56, and a magnetic navigation system 58 comprising opposed magnet units60 and 62 on opposite sides of the patient bed operated by a processor64 and controlled by controls 66 adjacent the patient 56. An imagingsystem 68, such as a x-ray imaging on a C-arm, displays images of theoperating region on a monitors 70 in the procedure room 50. Theinterface system of the present invention provides a convenient way fora physician to operate the magnetic navigation system 58 to control thedistal end of a medical device in the operating region inside thepatient's body.

The interface includes a display on, for example, an LCD monitor 72, anda mouse 74 in the procedure room 50, a processor 76, a display on, forexample, monitor 78, a key board 80, and a mouse 82 in the control room54. Additional displays on monitors 86 and 88 can be provided in theprocedure room 50 which integrate images from the imaging system 68 withthe interface. One or more additional monitors 90 can be provided in thecontrol room so that the images are available in the control room aswell. The monitors 72 and 78 preferably display a multi-pane display.

In a first preferred embodiment, as shown in FIGS. 3 the display 100 onthe monitors 72 and 78, includes a menu bar 102, a tool bar 104, a 3-Ddisplay pane 106, a status area 108, a 2-D anatomical control pane 110,a point navigation control pane 112, and a vector navigation controlpane 114, and a bull's eye navigation control pane 116. Of course thedisplay 100 could include additional panes or fewer panes or differentpanes. An example of a display in accordance with this invention isshown in FIG. 4.

A 3-D display pane 106 in accordance with this invention is shown inFIG. 5. The display preferably includes a three-dimensionalrepresentation 120 of the patient orientation. As shown in FIG. 5 thisrepresentation 120 may be a representation of a horizontal gridcorresponding to the surface of the patient support 56. Alternatively,the may be a three dimensional representation of an idealized patient,or of the patient support 56. A coordinate system 122 is optionallyincluded in the representation to facilitate the physician'sunderstanding of the orientation. In the first preferred embodiment, thecoordinate system 122 comprises a longitudinal axis 122 x, which mightfor example be colored blue, a horizontal axis 122 y, which might forexample be colored red, and a anterior-posterior axis 122 z, whichmight, for example be colored green. The pane 106 preferably alsoincludes a subpane 124 that displays three dimensional representation ofthe operating region. In this first preferred embodiment thisrepresentation is an transparent, three dimensional idealizedrepresentation of the portion of the patient's body in which theprocedure is taking place, e.g. a human heart as shown in FIG. 5. Tofacilitate the user's interpretation of the image, the image may bedisplayed over a horizontal backing grid. Instead of an idealizedrepresentation of the procedure site, the image could be an actualpreoperative image, or an actual current image. A coordinate system 126is optionally included in the representation to facilitate the user'sunderstanding of the orientation. In the first preferred embodiment, thecoordinate system 126 comprises a longitudinal axis 126 x, parallel tothe direction as axis 122 x, and which may similarly be colored blue, ahorizontal axis 126 y, parallel to the direction of axis 122 y, andwhich may similarly be colored red, and a anterior-posterior axis 126 z,parallel to the direction of axis 122 z, and which may similarly becolored green.

The tool bar 104 includes a 3D tool bar 128 with controls forcontrolling the 3-D display pane 106. In this first preferredembodiment, these controls include a translation button 130, amagnification button 132, a rotation button 134, a point selectionbutton 136, a point centering button 138, an image autorotate button140, a swap button 142, and an image capture button 144. These buttonsare preferably “virtual buttons”, i.e., they are elements on the displaywhich the user can operate by pointing a cursor and clicking.

A view selection menu bar 146 is also provided on the 3D tool bar 128.The view selection menu 146 has an arrow that can be operated to dropdown a menu of views to display in the pane 106. These preferablyinclude cranial, caudal, anterior, posterior, left and right, as well asone or more user defined views. Of course other standard views could beprovided depending upon the procedures for which the interface is used.

The translation button 130 can be actuated to enter the viewpointtranslation mode by pointing the cursor to the button and clicking. Inthe viewpoint translation mode, the cursor might change in appearance,for example to a shape corresponding to the icon on the button 130. Inthis mode the view point can be changed by grabbing the image byclicking when the cursor is on the image, and dragging the cursor tomove the image and thus the viewpoint in any direction. The cursor canbe moved using mouse 74 or 82. This preferably also causes acorresponding translation of the view point of the image in the subpane124.

The magnification button 132 can be operated to enter the magnificationor zoom mode by pointing the cursor to the button and clicking, forexample with mouse 74 or 82. In the zoom mode the cursor might change inappearance, for example to a shape corresponding to the magnifying glassicon on the button 132. In this mode the magnification of the patientreference image 120 can be accomplished by grabbing the image bypointing the cursor and clicking, and dragging the cursor downwardlyand/or to the right to increase the magnification, or upwardly or to theleft to decrease the magnification. Changing the size of the patientreference image preferably also does not change the size of theprocedure site reference image.

The rotation button 134 can be operated to enter the image rotation modeby pointing the cursor to the button and clicking, for example withmouse 74 or 82. In the image rotation mode the cursor might change inappearance, for example to a shape corresponding to the shape on thebutton 134. In this mode the image can be rotated by grabbing the imageby pointing the cursor and clicking, and dragging the cursorhorizontally to rotate the view point of the image about a generallyvertical axis, and vertically to rotate the view point about a generallyhorizontal axis. Of course the image can be dragged both horizontallyand vertically to rotate the axis about a diagonal axis. Rotating thepatient reference image preferably also rotates the procedure sitereference image, so that these two images always have the sameviewpoint.

The point select button 136 can be operated to enter the point selectionmode by pointing the cursor to the button and clicking, for example withmouse 74 or 82. In the point selection mode the cursor might change inappearance, for example to a shape corresponding to the shape on thebutton 136. In this mode a point in the image 120 can be selected bymoving the cursor over a point on image and clicking, for example withmouse 74 or 82. The selection of the point causes the point to beidentified on the point navigation pane 112, as described in more detailbelow.

The point center button 138 can be operated to enter the point selectionmode by pointing the cursor to the button and clicking, for example withmouse 74 or 82. In the point center mode the cursor might change inappearance, for example to a shape corresponding to the shape on thebutton 138. In this mode the view point for the image 120 can becentered upon a selected point by moving the cursor over a point onimage and clicking, for example with mouse 74 or 82.

The autorotation button 140 can be operated to enter the autorotationmode by pointing the cursor to the button and clicking, for example withmouse 74 or 82. In the autorotation mode the cursor might change inappearance, for example to the shape corresponding to shape on therotation button. In this mode the viewpoint for the image rotatesautomatically horizontally to the left. The direction of the rotationcan be changed by pointing the cursor on the image and clicking anddragging in the desired new direction of rotation.

The image swap button 142 can be operated to swap the images displayedin the main pane 106 and in the subpane 124 by pointing the cursor tothe button and clicking, for example with mouse 74 or 82.

The image capture button 144 can be operated to enter the image capturemode by pointing the cursor to the button, and clicking, for examplewith mouse 74 or 82. This opens a box that allows the user to save theimage on the pane 106 for future reference.

The interface preferably displays a visual indicator of the desiredorientation for the distal end of the medical device. In this firstpreferred embodiment, this indicator is an arrow 150, whose shaft isaligned with the desired orientation, with a large conical head pointingin the desired direction. The arrow 150 is preferably a distinctivecolor such as green. The interface preferably also displays a visualindicator of the current orientation of the distal end of the medicaldevice. In this first preferred embodiment, this indicator is an arrow152, whose shaft is aligned with the current orientation of the distalend of the medical device, with a larger conical head pointing in thedesired direction.

A localization system could be provided for determining the currentposition and orientation of the distal end of the medical device. Animage representative of the distal end of the medical device can then begenerated and displayed in the pane 106. There are numerous method forlocalizing the distal end of the medical device, for exampletransmitting magnetic signals between one or more reference locationsand the medical device, x-ray image processing, ultrasound localization,or electric potential localization.

In the first preferred embodiment, the interface is adapted for use witha magnetic navigation system that operates by generating a magneticfield of selected direction in the operating region, which causes amagnetically responsive element associated with the distal end of themedical device to generally align with the applied field. Because of thephysical properties of the catheter, limitations in the strength of theapplied field, and the conditions in the procedure site, the distal endof the medical device may not align precisely with the applied magneticfield. While the difference between the applied magnetic field and theactual direction of the distal end of the medical device can beaccounted for through modeling or a look-up table, in the firstpreferred embodiment the arrow 150 representing the desired orientationmay represent the desired direction of the applied magnetic field,rather than the desired direction of the medical device itself.Similarly, the arrow 152 representing the current orientation mayrepresent the direction of the magnetic field to currently beingapplied, rather than the actual direction of the device itself. However,the differences between the actual direction of the medical device andthe applied magnetic field can be characterized by equation or anempirically determined look-up table, or localization of the device canbe provided so that even when used with a magnetic navigation system,the arrow 150 represents the actual desired orientation of the medicaldevice, and arrow 152 represents the actual current direction.

To help visualize the three-dimensional direction of the indicator, thearrow 150 can be surrounded with an “umbrella” 154—a shape or surfacesurrounding the arrow so that its direction and orientation can be moreeasily visualized. One implementation of the umbrella 154 is as a wireframe hemisphere. In addition to improving the visualization of thedirection of the arrow 150, the umbrella 154 can be used to selectionthe orientation of the arrow 150. When the cursor hovers over thesurface of the umbrella, the cursor can change appearance, for exampleto resemble the rotation icon on button 134. The direction of the arrow150 can be changed by rotating the hemisphere by pointing the cursor tothe hemisphere, clicking, and dragging the cursor in the desireddirection of rotation. In addition the arrow 150 and hemisphere 154 canbe configured so that when the cursor hovers over the root of the arrow150, the cursor can change in appearance, for example to resemble thetranslation icon on button 130. The position of the root of the arrow150 can be changed by clicking the cursor and dragging the cursor in thedesired direction of movement.

In the first preferred embodiment, the interface includes displays ofthe fluoroscopic images of the operating region, with the arrow 150superposed thereon. For example, as shown in FIGS. 6A and 6B, theimaging system 68 can provide biplanar images of the operating region,and the arrow 150 and umbrella 154 provided on each image. These imagescould be displayed on monitors 86 and 88 in the procedure room 50, andon monitor 90 in the control room 52. Preferably, the user can changethe direction of the arrow 150 on these images as well by rotating andtranslating the arrow and umbrella as described above.

The display 100 of the interface preferably also includes a status area108, where, as shown in FIG. 7, a text, graphic, or combination text andgraphic message of the status of the interface can be displayed to theuser. These messages can be colored coded for example to convey animmediate impression of the importance or significance of the messagedisplayed.

While the orientation of the distal end of the medical device can bemanipulated directly on the pane 106, for example by manipulating theumbrella 154, the display 100 of the interface preferably includes atleast one pane to aid the user in selecting the desired orientation forthe medical device. In this first preferred embodiment there are severalpanes that provide alternative methods for the user to select thedesired orientation for the distal end of the medical device. Thesepanes include representations of the orientation of the arrow 150 whichare constantly updated, so that use of one pane to change the desireddirection of the medical device, causes all of the other panes toupdate, to facilitate the use of any of the panes to adjust theorientation of the arrow 150 representing the desired new orientation ofthe medical device.

One such pane to aid the user in selecting the desired orientation forthe medical device is the 2-D anatomical pane 110, which allows the userto select the desired orientation of distal end of the medical device asindicated by the arrow 150 by adjusting the direction in one or moreplanes through the patient. As shown in FIG. 8A, the pane 110 allows theuser to change the direction of the arrow 150 in at least one plane, andpreferably at least two planes and more preferably at least the planes.These planes are preferably, but not necessarily, mutuallyperpendicular. While adjustment in two planes is sufficient to specifyany direction, providing adjustment in three planes makes it easier fora user to select the desired direction for the arrow 150. In this firstpreferred embodiment, the arrow can be rotated in the coronal or frontalplane (i.e., about an anterior-posterior axis), the median or saggitalplane (i.e., about a horizontal axis), and the horizontal or transverseplane (i.e., about a longitudinal axis).

As shown in FIG. 8A the pane 110 can have three graphic displays 160,162 and 164, corresponding to the three planes of rotation. Graphicdisplay 160 contains a graphic depiction of the coronal or frontal plane(i.e., an caricature image of a patient's body in the coronal or frontalplane), with an indicator 166 that indicates the orientation of thearrow in the coronal or frontal plane, and virtual buttons 168 and 170for moving the indicator 166 (and thus the arrow 150) clockwise orcounterclockwise in the coronal or frontal plane abut theanterior-posterior axis. In this first preferred embodiment, indicator166 is actually a projection of the arrow 150 in the plane, and thus thelength of the indicator 166 is indicative of the orientation. Thevirtual buttons 168 and 170 can be operated with a cursor for examplewith the mouse 74 or 82 or the keyboard 80, to point and click thebutton and move the indicator 166 and thus the arrow 150, in the desireddirection. Display 162 contains a graphic depiction of the median orsaggital plane (i.e., a caricature image of a patient's body in themedian or saggital plane), with an indicator 172 indicating thedirection of the arrow 150 in the median or saggital plane, and virtualbuttons 174 and 176 for moving the indicator 172 (and thus the arrow150) clockwise or counterclockwise in the coronal or frontal plane. Inthis first preferred embodiment, indicator 172 is actually a projectionof the arrow 150 in the plane, and thus the length of the indicator 172is indicative of the orientation. The virtual buttons 174 and 176 can beoperated with a cursor for example with the mouse 74 or 82 or thekeyboard 80, to point and click and move the indicator 172 and thus thearrow 150, in the desired direction. Display 164 contains a graphicdepiction of the horizontal or transverse plane (i.e., a caricatureimage of a patient's body in the horizontal or transverse plane), withan indicator 178 indicating the direction of the arrow in the horizontalor transverse plane, and virtual buttons 180 and 182 for moving theindicator 178 (and thus the arrow 150) clockwise or counterclockwise inthe horizontal or transverse plane. The virtual buttons 180 and 182 canbe operated with a cursor for example with the mouse 58 or 66 or thekeyboard 64, to point and click to move the indicator 178, and thus thearrow 150 in the desired direction.

The pane 110 also includes a menu 184 to select the increment of changein direction upon operating the buttons 168 and 170, 174 and 176, and180 and 182. The user can select the incremental change from 1 degree, 2degrees, 3 degrees, 5 degrees, 10 degrees with a cursor for example withthe mouse 74 or 82 or the keyboard 80, to point and click to select thedesired increment.

An alternate implementation of the pane 110′ is shown FIG. 8B. Incontrast to FIG. 8A where pane 110 allows movement of the arrow 150relative to the coronal or frontal plane, the median or saggital plane,and the horizontal or transverse plane, in FIG. 8B the pane 110′ allowsmovement of the arrow 150 relative to the right anterior oblique plane,the left anterior oblique plane, and the transverse plane. As shown inFIG. 8B the pane 110 can have three graphic displays 160′, 162′ and164′, corresponding to the three planes of rotation. Graphic display160′ contains a graphic depiction of the right anterior oblique plane(i.e., an caricature image of patient's body or part of the patient'sbody in the RAO plane), with an indicator 166′ that indicates theorientation of the arrow in the coronal or frontal plane, and virtualbuttons 168′ and 170′ for moving the indicator 166′ (and thus the arrow150) clockwise or counterclockwise in the left anterior oblique plane.In this first preferred embodiment, indicator 166′ is actually aprojection of the arrow 150 in the plane, and thus the length of theindicator 166′ is indicative of the orientation. The virtual buttons168′ and 170′ can be operated with a cursor for example with the mouse74 or 82 or the keyboard 80, to point and click the button and move theindicator 166′ and thus the arrow 150, in the desired direction. Display172′ contains a graphic depiction of the left anterior oblique plane(i.e., a caricature image of a patient's body or portion of thepatient's body in the LAO plane), with an indicator 172′ indicating thedirection of the arrow 150 in the median or saggital plane, and virtualbuttons 174′ and 176′ for moving the indicator 172′ (and thus the arrow150) clockwise or counterclockwise in the coronal or frontal plane. Inthis first preferred embodiment, indicator 172′ is actually a projectionof the arrow 150 in the plane, and thus the length of the indicator 172′is indicative of the orientation. The virtual buttons 174′ and 176′ canbe operated with a cursor for example with the mouse 74 or 82 or thekeyboard 80, to point and click and move the indicator 172′ and thus thearrow 150, in the desired direction. Display 164′ contains a graphicdepiction of the horizontal or transverse plane (i.e., a caricatureimage of a patient's body or a portion of the patient's body in thehorizontal or transverse plane), with an indicator 178′ indicating thedirection of the arrow in the horizontal or transverse plane, andvirtual buttons 180′ and 182′ for moving the indicator 178′ (and thusthe arrow 150) clockwise or counterclockwise in the horizontal ortransverse plane. The virtual buttons 180′ and 182′ can be operated witha cursor for example with the mouse 58 or 66 or the keyboard 64, topoint and click to move the indicator 178′, and thus the arrow 150 inthe desired direction.

The pane 110′ also includes a menu 184′ to select the increment ofchange in direction upon operating the buttons 168′ and 170′, 174′ and176′, and 180′ and 182′. The user can select the incremental change from1 degree, 2 degrees, 3 degrees, 5 degrees, 10 degrees with a cursor forexample with the mouse 74 or 82 or the keyboard 80, to point and clickto select the desired increment.

Another pane to aid the user in selecting the desired orientation forthe arrow 150 and thus for the medical device is a point navigation pane112. As shown in FIG. 9, the point navigation pane 112 includes a groupmenu table 200 containing information about one or more groups of pointsthe user identifies. The group menu table 200 includes a column 200 awith a color indicator for indicating the color corresponding to thepoints in the group. All points in the group will be indicated with amark in the indicated color. The menu table 200 further includes acolumn 200 b entitled “Group Name” with the name of a stored group ofpoints. The menu table 200 further includes a column 200 c entitled “3D”which indicates whether the group of points is visible on the 3D displayin pane 106 (“show”) or not visible on the 3D display in the pane(“hide”). Finally, the table comprises a column 200 entitled “Fluro”which indicates whether the group of points is visible on the 3D displayin pane 106 (“show”) or not visible on the 3D display in the pane(“hide”).

A “new” button 202, a “delete” button 204, and an “edit” button 206 areassociated with the menu table 200. The buttons 202, 204, and 206 arepreferably “virtual” buttons, i.e. portions of the display on which theuser points the cursor and clicks, for example with mouse 74 or 82, orkeyboard 82. The new button 202 can be operated by pointing and clickingwith the cursor using the mouse 74 or 82 or keyboard 80, and allows theuser to create a new group in the menu table 200. Operating the newbutton 202 opens a box that allows the user to select the colorindicator in column 200 a, select the name of the group in column 200 b,select the display properties in column 200 c between “show” and “hide”to determine whether the points will appear on the 3D panel 110, andselect the display properties in column 200 d, between “show” and “hide”to determine whether the points will appear on the fluoroscope displays(monitors 86, 88, and 90). The delete button 204 can be operated bypointing and clicking with the cursor using the 74 or 82, or keyboard82, and allows the user to delete the group or groups that the userhighlighted in the menu table 114, using the mouse 74 or 82, or keyboard82. The edit button 206 can be operated by pointing and clicking withthe cursor using the mouse 74 or 82, or keyboard 82, and allows the userto edit the group that the user highlighted in the menu table 200 usingthe 74 or 82, or keyboard 82. Operating the edit button 206 opens a boxthat allows the user to change the color indicator in column 200 a,change the name of the group in column 200 b, change the displayproperties in column 200 c between “show” and “hide” to determinewhether the points will appear on the 3D panel 110, and change thedisplay properties in column 200 d, between “show” and “hide” todetermine whether the points will appear on the fluoroscope displays(monitors 86, 88, and 90).

The pane 112 also includes a point menu table 208. The menu table 208includes a column 208 a, entitled “id” for an identification codeassigned by the system to a particular point (in the first preferredembodiment the system assigns an id from A to ZZ). The menu table 208further includes a column 208 b, entitled “point name” for the name ofthe point. Finally, the menu table 208 includes a third column 208 centitled “group” for the name of the group to which the point isassigned. A display control is provided adjacent the point menu table208 for selection the points to display in the point menu table 208. Asshown in FIG. 9, the display control can comprise radio buttons 210 and212, which allow the user to specify “all groups” or “selected group” ,respectively, so that the user can identify whether to display thepoints in “all groups” or just the points a selected group “selectedgroup” in the menu table 208.

An “edit” button 214, a “delete” button 216, a “group” button 218, and a“vector” button 220 are associated with the menu table 208. The buttons214, 216, 218, and 220 are preferably “virtual” buttons on the displaythat can be operated by pointing the cursor and clicking, for examplewith mouse 74 or 82, or keyboard 80. The user can select a point on themenu table 200 by pointing with the cursor and clicking, using the muse74 or 82, or the keyboard 80. The edit button 214 can be operated bypointing and clicking with the cursor using the mouse 74 or 82 orkeyboard 80, and allows the user to edit the selected point. Operatingthe edit box opens a box that allows the user to change the name of theselected point in column 208 b, and the group to which the point iscolumn 208 c. The delete button 216 can be operated by pointing andclicking with the cursor using the mouse 74 or 82 or keyboard 80, andallows the user to delete the selected point. The group button 218 canbe operated by pointing and clicking with the cursor using the mouse 74or 82 or keyboard 80, and allows the user to change the group to whichthe selected point is associated. The vector button 220 can be operatedby pointing and clicking with the cursor using the mouse 74 or 82 orkeyboard 80, and allows the user to set the orientation of the arrow 150to the orientation associated with a point selected on the menu table208 using the mouse 74 or 82 or keyboard 80. This automatically updatesthe display of arrow 150 in the other panes. Thus a user who wants tonavigate back to a stored point can recall the direction associated withthat point, facilitating the return to the point. However that directionmay also be useful in navigating to another point.

Another such pane to aid the user in selecting the desired orientationfor the medical device is a vector navigation pane 114. The vectornavigation pane 114 allows the user to use predetermined directions, tostore and use new directions, and to recall and use previously useddirections. The vector navigation pane 114 includes a section 222 forrecalling and using predetermined directions; a direction vector storageand recall section 224; and a direction history and recall section 226.The section 222 for recalling and using predetermined directionsincludes a “preset list” pick menu 228 for selecting a particular set ofpredetermined directions, and a “direction” pick menu 230 for selectinga particular direction from the selected set. A set of possible “presetlist” and “direction” entries for the pick menus 228 and 230 is shown inTable 1. The user can select from the “preset list” and “direction” pickmenus using the mouse 74 or 82 or keyboard 80. TABLE 1 Possible PresetLists and Directions Cardinal Superior Cardinal Inferior CardinalAnterior Cardinal Posterior Cardinal Left Cardinal Right Cardinal RAOCardinal LAO Deflection From 0 to 175° in 15° increments

The direction vector storage and recall section 224 includes a vectormenu table 232, and associated “store” button 234, “edit” button 236,“delete” button 238. The buttons 234, 236, and 238 are preferablyvirtual buttons, or portions of the display to which the cursor can bepointed and clicked, for example with the mouse 74 or 82, or thekeyboard 80. The “store” button 234 can be operated by pointing andclicking with the cursor using the mouse 74 or 82 or keyboard 80, andallows the user to store the current direction under a user selectedname on the vector menu table 232. Operating the store button 234 opensa box that allows the user to input a name. The user can selected astored direction from the menu table 232 by pointing to the name withthe cursor, and clicking, using the mouse 74 or 82, or keyboard 80. The“edit” button 236 can be operated by pointing and clicking with thecursor using the mouse 74 or 82 or keyboard 80, and allows the user toedit the name of a selected direction. The “delete” button 238 can beoperated by pointing and clicking with the cursor using the mouse 74 or82 or keyboard 80, and allows the user to delete a selected direction.The history section 226 includes virtual forward and back buttons 240and 242. The forward and back buttons 240 and 242 can be operated bypointing the cursor and clicking using mouse 74 or 82 or keyboard 80.The buttons 240 and 242 allow the user to set the orientation of thearrow 150 to one of the previously selected directions, which areautomatically stored. In the first preferred embodiment, the systemautomatically stores the last ten directions, and the user can scrollbackward and forward through these directions with the buttons 240 and242. The appearance of the buttons 240 and 242 changes (e.g. grays out)when the there is no further stored directions.

The bull's eye navigation pane 116 includes a circular screen 250, andan “apply” button 252. The pane 116 also includes a scale menu 254,which in the first preferred embodiment allows the user to select thescale of the screen 250 from 15, 20, 45, 60, and 90 degrees. The usercan select the desired scale for the circular screen 250 by pointing thecursor and clicking, using the mouse 74 or 82 or keyboard 80. The pane116 may also include a display control section 256 with “Hide” and“Show” radio buttons 258 and 260. These buttons determine whether thecircular screen 250 is projected onto the other displays, specificallythe 3D display of pane 106 and the fluoroscopic images from the imagingsystem displayed on the monitors 86, 88, and 90. FIG. 12 shows one ofthe biplane imaging displays with the screen 250 projected thereon. Thedisplay control section 256 also includes RAO (right anterior oblique)and LAO (left anterior oblique) selection buttons 262 and 264, whichorient the screen 250 so that the top of the screen is up in whicheverof the two views is selected. As shown in FIG. 11, markers 262 and 264are provided on the circular screen 250, to help the user interpret theorientation of the circular screen 250 on the 3D pane 106 and the on theRAO and LAO views. The marker 262 might be blue and the marker 264 mightbe red.

The user can set the base direction the navigation pane 116 by operatingthe “apply” button 252 by pointing at the button with a cursor andclicking, using mouse 74 or 82 or keyboard 80. The sets the currentdirection as the direction though the center of the screen 250. The usercan then specify a direction for the arrow 150 by selecting a point onthe screen 250, by pointing with the cursor and clicking, using mouse 74or 82, or keyboard 80. As shown in FIG. 11, the screen 250 has verticaland horizontal cross hairs 266 and 268, and a plurality of radiallyextending markers 270, at 30 degree intervals. There are a plurality ofconcentric circular markers 272 representing regular angular intervals(10 degree intervals in the first preferred embodiment), with specifiedintervals (30 degree intervals in the first preferred embodiment)indicated by bold markers 272 a. The circular screen 272 actuallyrepresents a hemisphere of space. The screen allows the user to orientthe arrow 150 at a number of points to draw radial and circular lines.

The toolbar 104 preferably also includes an indicator 280, an applybutton 282, a reduce button 284, and an angle indicator 286. Theindicator 280 indicates when the interface is connected to the magneticnavigation system. Of course if some other system for orienting thedistal end of the medical device is used, a suitable indicator can beprovided. The apply button 282 and the reduce button 284 are preferablyvirtual buttons which are operated by pointing the cursor and clicking,for example with mouse 74 or 82, or keyboard 80. Operating the applybutton 282 causes the magnetic navigation system to apply a magneticfield to orient the distal end of the medical device in the orientationof the arrow 150. Operating the reduce button 284 causes the magneticnavigation system to “turn off” the magnetic field. The indicator 286indicates the angular difference between the previously applied magneticfield and the orientation of arrow 150. Of course rather than discretenavigation, in which the arrow 150 is successively oriented and themagnetic field applied, the interface could be adapted to operate in acontinuous navigation mode in which the field is automatically appliedin the direction of arrow 150

Operation

In operation the user can visualize the current direction of the devicerepresented by arrow 154 and the desired new direction for the devicerepresented by arrow 150, on the 3-D pane 106 or on the x-ray images onmonitors 86, 88, and 90. The user can selected the orientation of thearrow 150 in a number of ways using panes 110, or 112, or 114, or 116.

The user can select the orientation of arrow 150 on pane 110 by clickingon buttons 168 and 170, 174 and 176, and 180 and 182, to move the arrow150 in each of the coronal or frontal plane, the median or saggitalplane, and the horizontal or transverse plane to move the arrow.Alternatively, the user can select the orientation of arrow 150 by usingthe pane 112. The user selects a point on the menu table 208 by pointingand clicking with the cursor, and then operating the vector button 220by pointing and clicking with the cursor. This sets the orientation ofarrow 150 to the orientation associated with point selected.Alternatively, the user can select the orientation of arrow 150 usingthe pane 114. The user can select a stored orientation by selecting acategory on menu 228, and a direction on menu 230. The user can select auser-stored direction by selecting a direction vector from the menutable 232. The user can select a previously used direction by using thebuttons 240 and 242 to recall one of the last previously used direction.Finally, the user can select an orientation by picking a point on ascreen 250.

Once the direction of the arrow 150 is selected, the navigation systemcan be operated by operating the apply button 282. This can operate amagnetic navigation system to apply a field in the direction 150, or itcan operate a magnetic navigation system to apply a field to cause themedical device to align in the direction 150, either by using feedbackof the catheter position or by calculating or using a look-up table toaccount for the properties of medical device.

In second preferred embodiment, as shown in FIGS. 14 the display 100′ onthe monitors 72 and 78, includes a menu bar 302, tool bars 304, a 3-Ddisplay pane 306, a 2-D anatomical control pane 308, a point navigationcontrol pane 310, a vector navigation control pane 312, and a bull's eyenavigation control pane 314, an advancer control pane 316, and a titleblock and device selection pane 318. Of course the display 100′ couldinclude additional panes or fewer panes or different panes. An exampleof a display in accordance with this invention is shown in FIG. 15.

A 3-D display pane 306 in accordance with this invention is shown inFIG. 15. The display preferably includes a three-dimensionalrepresentation of the patient orientation. As shown in FIG. 15 thisrepresentation may be a representation of a horizontal gridcorresponding to the surface of the patient support 56. Alternatively,the may be a three dimensional representation of an idealized patient,or of the patient support 56. The pane 306 preferably also includes asubpane 324 that displays three dimensional representation of theoperating region. In this preferred embodiment this representation is antransparent, three dimensional idealized representation of the portionof the patient's body in which the procedure is taking place, e.g. ahuman heart as shown in FIG. 15. To facilitate the user's interpretationof the image, the image may be displayed over a horizontal backing grid.Instead of an idealized representation of the procedure site, the imagecould be an actual preoperative image, or an actual current image. Acoordinate system is optionally included in the representation tofacilitate the user's understanding of the orientation.

The tool bar 304 includes a 3D tool bar 328 with controls forcontrolling the 3-D display pane 306. In this second preferredembodiment, these controls include a screen manipulation button 330, agrid button 332, a display selector button 334, a constellation button336; a point centering button 338, a zoom in button 340, a zoom outbutton 342, and an image capture button 344. These buttons arepreferably “virtual buttons”, i.e., they are elements on the displaywhich the user can operate by pointing a cursor and clicking.

A view selection menu bar 346 is also provided on the 3D tool bar 328.The view selection menu 346 has an arrow that can be operated to dropdown a menu of views to display in the pane 306. These preferablyinclude cranial, caudal, anterior, posterior, left and right, as well asone or more user defined views. Of course other standard views could beprovided depending upon the procedures for which the interface is used.

The screen manipulation button 330 can be actuated (for example by rightclicking) to display a plurality of screen manipulation options for thecursor. For example, the user can select among a plurality of cursormodes to translate the image on the display 306, to rotate the image onthe display, etc., by clicking and dragging the image. The appearance ofthe cursor on the display 306 preferably changes to cue the user as tothe particularly screen manipulation mode in effect. In the translationmode, the cursor might change in appearance, for example to a shapecorresponding to the icon on the button 330. In this mode the view pointcan be changed by grabbing the image by clicking when the cursor is onthe image, and dragging the cursor to move the image and thus theviewpoint in any direction. The cursor can be moved using mouse 74 or82. This preferably also causes a corresponding translation of the viewpoint of the image in the subpane 324.

The grid button 332 can be clicked to show and hide the grid lines onthe display 306.

The display selector button 334 allows the user to select the format ofthe display 306. The user can click on the button to cause a menu oficons depicting various formats to drop down. The user then simplyselects the desired format, for example including the subpane 324 (asshown) or removing the subpane 324.

The display constellations button 336 can be operated to toggle betweena display in which points on the display 306 are shown as part of agroup or constellation (e.g. FIG. 19) by pointing the cursor to thebutton and clicking, for example with mouse 74 or 82.

The point center button 338 can be operated to enter the point selectionmode by pointing the cursor to the button and clicking, for example withmouse 74 or 82. In the point center mode the cursor might change inappearance, for example to a shape corresponding to the shape on thebutton 338. In this mode the view point for the image can be centeredupon a selected point by moving the cursor over a point on image andclicking, for example with mouse 74 or 82.

The zoom in button 340 allows the user to click to enlarge the image onthe display 306, and the zoom out button 342 allows the user to click toreduce the image on the display 306 The zoom in button 340 and the zoomout button 342 can be operated to enter the magnification or zoom modeby pointing the cursor to the button and clicking for example with mouse74 or 82. In the zoom mode the cursor might change in appearance, forexample to a shape corresponding to the magnifying glass icon with a “+”for zoom in, and a “−” for zoon out. In this mode the magnification ofthe image can be accomplished by grabbing the image by pointing thecursor and clicking, and dragging the cursor downwardly and/or to theright to increase the magnification, or upwardly or to the left todecrease the magnification. Changing the size of the patient referenceimage 306 preferably also does not change the size of the procedure sitereference image. 324

The image capture button 344 can be operated to enter the image capturemode by pointing the cursor to the button, and clicking, for examplewith mouse 74 or 82. This opens a box that allows the user to save theimage on the pane 306 for future reference.

The interface preferably displays a visual indicator of the desiredorientation for the distal end of the medical device. In this preferredembodiment, this indicator is an arrow 350, whose shaft is aligned withthe desired orientation, with a large conical head pointing in thedesired direction. The arrow 350 is preferably a distinctive color, e.g.green. The interface preferably also displays a visual indicator of thecurrent orientation of the distal end of the medical device. In thispreferred embodiment, this indicator is an arrow 352, whose shaft isaligned with the current orientation of the distal end of the medicaldevice, with a larger conical head pointing in the desired direction.The arrow 352 is preferably a distinctive color, different from thearrow 350, e.g. yellow.

A localization system could be provided for determining the currentposition and orientation of the distal end of the medical device. Animage representative of the distal end of the medical device can then begenerated and displayed in the pane 306. There are numerous method forlocalizing the distal end of the medical device, for exampletransmitting magnetic signals between one or more reference locationsand the medical device, x-ray image processing, ultrasound localization,or electric potential localization.

In the preferred embodiment, the interface is adapted for use with amagnetic navigation system that operates by generating a magnetic fieldof selected direction in the operating region, which causes amagnetically responsive element associated with the distal end of themedical device to generally align with the applied field. Because of thephysical properties of the catheter, limitations in the strength of theapplied field, and the conditions in the procedure site, the distal endof the medical device may not align precisely with the applied magneticfield. While the difference between the applied magnetic field and theactual direction of the distal end of the medical device can beaccounted for through modeling or a look-up table, in the preferredembodiment the arrow 350 representing the desired orientation mayrepresent the desired direction of the applied magnetic field, ratherthan the desired direction of the medical device itself. Similarly, thearrow 352 representing the current orientation may represent thedirection of the magnetic field to currently being applied, rather thanthe actual direction of the device itself. However, the differencesbetween the actual direction of the medical device and the appliedmagnetic field can be characterized by equation or an empiricallydetermined look-up table, or localization of the device can be providedso that even when used with a magnetic navigation system, the arrow 350represents the actual desired orientation of the medical device, andarrow 352 represents the actual current direction.

As in the first preferred embodiment, in the second preferredembodiment, the interface includes displays of the fluoroscopic imagesof the operating region, with the arrow 350 superposed thereon. Forexample, as shown in FIGS. 6A and 6B, the imaging system 68 can providebiplanar images of the operating region, and the arrow 350 on eachimage. These images could be displayed on monitors 86 an 88 in theprocedure room 50, and on monitor 90 in the control room 52. Preferably,the user can change the direction of the arrow 150 on these images byrotating and translating the arrow as described above.

While the orientation of the distal end of the medical device can bemanipulated directly on the pane 306, the display 100′ of the interfacepreferably includes at least one pane to aid the user in selecting thedesired orientation for the medical device, and thus of the arrow 350.In this preferred embodiment there are several panes that providealternative methods for the user to select the desired orientation forthe distal end of the medical device. These panes includerepresentations of the orientation of the arrow 350 which are constantlyupdated, so that use of one pane to change the desired direction of themedical device, causes all of the other panes to update, to facilitatethe use of any of the panes to adjust the orientation of the arrow 350representing the desired new orientation of the medical device.

One such pane to aid the user in selecting the desired orientation forthe medical device is the 2-D anatomical pane 308, which allows the userto select the desired orientation of distal end of the medical device asindicated by the arrow 350 by adjusting the direction in one or moreplanes through the patient. As shown in FIG. 15, the pane 310 allows theuser to change the direction of the arrow 350 in at least one plane, andpreferably at least two planes and more preferably at least the planes.These planes are preferably, but not necessarily, mutuallyperpendicular. While adjustment in two planes is sufficient to specifyany direction, providing adjustment in three planes makes it easier fora user to select the desired direction for the arrow 350. In thispreferred embodiment, the arrow 350 can be rotated in the coronal orfrontal plane (i.e., about an anterior-posterior axis), the median orsaggital plane (i.e., about a horizontal axis), and the horizontal ortransverse plane (i.e., about a longitudinal axis).

As shown in FIGS. 15-19 the pane 308 can have three graphic displays360, 362 and 364, corresponding to the three planes of rotation. Asshown in FIGS. 15 and 16, the user can preferably select between ananatomy view (FIG. 15) or a whole body view (FIG. 16). Graphic display360 contains a graphic depiction of the coronal or frontal plane (e.g. acaricature image of the organ and/or operation region or a caricatureimage of a patient's body, in the coronal or frontal plane), with anindicator 366 that indicates the orientation of the arrow in the coronalor frontal plane, and virtual buttons 368 and 370 for moving theindicator 366 (and thus the arrow 350) clockwise or counterclockwise inthe coronal or frontal plane abut the anterior-posterior axis. In thispreferred embodiment, indicator 366 is actually a projection of thearrow 350 in the plane, and thus the length of the indicator 366 isindicative of the orientation. The virtual buttons 368 and 370 can beoperated with a cursor for example with the mouse 74 or 82 or thekeyboard 80, to point and click the button and move the indicator 366and thus the arrow 350, in the desired direction. Display 362 contains agraphic depiction of the median or saggital plane (i.e., a caricatureimage of a patient's body in the median or saggital plane), with anindicator 372 indicating the direction of the arrow 350 in the median orsaggital plane, and virtual buttons 374 and 376 for moving the indicator372 (and thus the arrow 350) clockwise or counterclockwise in thecoronal or frontal plane. In this preferred embodiment, indicator 372 isactually a projection of the arrow 350 in the plane, and thus the lengthof the indicator 372 is indicative of the orientation. The virtualbuttons 374 and 376 can be operated with a cursor for example with themouse 74 or 82 or the keyboard 80, to point and click and move theindicator 372 and thus the arrow 350, in the desired direction. Display364 contains a graphic depiction of the horizontal or transverse plane(i.e., a caricature image of a patient's body in the horizontal ortransverse plane), with an indicator 378 indicating the direction of thearrow in the horizontal or transverse plane, and virtual buttons 380 and382 for moving the indicator 378 (and thus the arrow 350) clockwise orcounterclockwise in the horizontal or transverse plane. The virtualbuttons 380 and 382 can be operated with a cursor for example with themouse 58 or 66 or the keyboard 64, to point and click to move theindicator 278, and thus the arrow 250 in the desired direction.

The pane 308 also includes a menu 384 to select the increment of changein direction upon operating the buttons 368 and 370, 374 and 376, and380 and 382. The user can select the incremental change from 1 degree, 2degrees, 3 degrees, 5 degrees, 10 degrees with a cursor for example withthe mouse 74 or 82 or the keyboard 80, to point and click to select thedesired increment.

Instead of using controls 368 and 370, 374 and 376, and 380 and 382, toincrementally move the indicators 366, 372. and 378, the user can simplypoint and click on the three graphic displays 360, 362 and 364 to movethe indicator to the selected point. Moving the indicators either withcontrols 368 and 370, 374 and 376, and 380 and 382, or by selectingpoints on the displays 366, 372, and 378, the user can selected thedirection of arrow 350.

Another pane to aid the user in selecting the desired orientation forthe arrow 350, and thus for the medical device, is point navigation pane310. As shown in FIG. 15, the point navigation pane 310 includes a groupmenu table 400 containing information about one or more groups of pointsthe user identifies. The group menu table 400 includes a column 400 afor an icon for identifying the arrangement of the group (a group ofpoints can be thought of as defining a shape, such as a circle, ellipse,or spline much the same way that stars for constellations of shapes). Anicon representing unorganized points is shown in FIG. 15, a iconrepresenting an ellipse “constellation” is shown in FIG. 19. Other typesof arrangements of points in a group, for example points on fittedcurve, and points on a spline, can be identified with different icons inthe column 400 a. A column 400 b, with the heading “Group Name” includesa color/shape identifier and a name for the group, e.g. “Group 1”. Inthis second preferred embodiment, a square of a color identifying thegroup is displayed in column 400 b, but the group could be identified insome other manner. All points in a group will be indicated with a markin the indicated color, as described in more detail below. The groupmenu table 400 further includes a column 400 c for a pick box for eachgroup for indicating whether the group should be shown on the bi-planefluoroscopic imaging screens (on monitors 86, 88, and 90), and a column400 d for a pick box for each group for indicating whether the groupshould be shown on the 3D display in pane 306.

The identified of points, groups of points, and constellations of pointswithin a group allows the user to simply identify a point or points andhave the interface determine the field direction to reach the point orpoints

The pane 310 also includes a point menu table 408. The menu table 408includes a column 408 a, for an identification symbol that indicates(preferably using color) the group to which the point belongs, a column408 b entitled “ID” that contains a code assigned by the system to aparticular point (in this second preferred embodiment the system assignsan ID sequentially from A to ZZ). The menu table 208 further includes acolumn 208 c, entitled “Point Name” for a user specified name of thepoint. The user can select a group by pointing the cursor on a group inthe group menu table 400, which causes the point menu table 408 todisplay each of the points in the selected group.

As a further aid to the user in selecting the desired orientation forthe medical device, vector navigation pick menus 428 and 430 areprovided on the toolbars 304. The pick menu 428 displays a “preset list”pick menu for selecting a particular set of predetermined directions,and the pick menu 430 displays a “direction” pick menu for selecting aparticular direction from the set selected in window 428. A set ofpossible “preset list” and “direction” entries for the pick menus 428and 430 is shown in Table 2. The user can select from the “preset list”and “direction” pick menus using the mouse 74 or 82 or keyboard 80.TABLE 2 Possible Preset Lists and Directions Cardinal Superior CardinalInferior Cardinal Anterior Cardinal Posterior Cardinal Left CardinalRight Cardinal RAO Cardinal LAO Deflection Increasing deflection from 0to 175° in 15° increments

Vector history buttons 432 and 434 are also provided on one of the toolbars 304 to aid the user in selecting the desired orientation for themedical device. The buttons 432 and 434 allow the user to movebackwardly and forwardly through an automatically stored list of appliedmagnetic field directions, in order to reapply a previously appliedmagnetic field. The buttons 432 and 434 allow the user to set theorientation of the arrow 350 to one of the previously selecteddirections, which are automatically stored. In the preferred embodiment,the system automatically stores the last ten directions, and the usercan scroll backward and forward through these directions with thebuttons 432 and 434. The appearance of the buttons 432 and 434preferably changes (e.g. grays out) when the there is no further storeddirections.

The interface can also include a vector storage and recall pane 312 tostore, recall, and use custom directions. The direction vector storageand recall pane 312 includes a vector menu table 436, and associated“store” button 438, “delete: button 440, and “edit” button 442. Thebuttons 438, 440, and 442 are preferably virtual buttons, or portions ofthe display to which the cursor can be pointed and clicked, for examplewith the mouse 74 or 82, or the keyboard 80. The “store” button 438 canbe operated by pointing and clicking with the cursor using the mouse 74or 82 or keyboard 80, and allows the user to store the current directionunder a user selected name on the vector menu table 436. Operating thestore button 438 allows the user to input a name for the storeddirection. The user can selected a previously stored direction from themenu table 436 by pointing to the name with the cursor, and clicking,using the mouse 74 or 82, or keyboard 80. The “edit” button 442 can beoperated by pointing and clicking with the cursor using the mouse 74 or82 or keyboard 80, and allows the user to edit the name of a selecteddirection. The “delete” button 440 can be operated by pointing andclicking with the cursor using the mouse 74 or 82 or keyboard 80, andallows the user to delete a selected direction.

The bull's eye navigation pane 314 includes a circular screen 450, andan “apply” button 452. The pane 314 also includes a scale menu 454,which in the preferred embodiment allows the user to select the scale ofthe screen 450 from 15, 20, 45, 60, 90, and 120 degrees. The user canselect the desired scale for the circular screen 250 by pointing thecursor at the scale menu 454, to display a list of scales, and selectingand clicking on the desired scale, using the mouse 74 or 82 or keyboard80. FIG. 17 shows the circular screen 450 on the display pane 306. Asshown in FIG. 17, markers 462 and 464 are provided on the circularscreen 450, to help the user interpret the orientation of the circularscreen 450 on the 3D pane 306 and the on the RAO and LAO views. Themarker 262 might be blue and the marker 264 might be red.

The user can set the base direction the navigation pane 116 by operatingthe “apply” button 452 by pointing at the button with a cursor andclicking, using mouse 74 or 82 or keyboard 80. This sets the currentdirection as the direction though the center of the screen 450. The usercan then specify a direction for the arrow 350 by selecting a point onthe screen 450, by pointing with the cursor and clicking, using mouse 74or 82, or keyboard 80. As shown in FIG. 17, the screen 450 has verticaland horizontal cross hairs 466 and 468, and a plurality of radiallyextending markers 470, at 30 degree intervals. There are a plurality ofconcentric circular markers 472 representing regular angular intervals(10 degree intervals in the preferred embodiment), with specifiedintervals (30 degree intervals in the preferred embodiment) indicated bybold markers 472 a. The circular screen 450 actually represents ahemisphere of space, and is represented as such with hemisphere 450′ ondisplay 306 in FIG. 17. The hemisphere 450′ includes markers 462′ and464′ corresponding to the markers 462 and 464 on circular screen 450.The screen 450 allows the user to orient the arrow 350 at a number ofpoints to draw radial and circular lines.

The toolbar 304 preferably also includes an indicator 480, an applybutton 482, a reduce button 484, and an angle indicator 486. Theindicator 480 allows the user to select among a “manual apply” mode, inwhich the user must affirmatively apply the selected field, an“automatic” mode in which the selected field direction is automaticallyapplied, and a “locked” mode in which the field cannot be appliedwithout changing the mode to either “manual apply” or “automatic”. Theapply button 482 and the reduce button 484 are preferably virtualbuttons which are operated by pointing the cursor and clicking, forexample with mouse 74 or 82, or keyboard 80. Operating the apply button482 when the interface is not in the automatic or locked modes causesthe magnetic navigation system to apply a magnetic field to orient thedistal end of the medical device in the orientation of the arrow 350.Operating the reduce button 484 causes the magnetic navigation system to“turn off” the magnetic field. The indicator 486 indicates the angulardifference between the previously applied magnetic field (arrow 352) andthe desired new orientation (arrow 350). Of course rather than discretenavigation, in which the arrow 350 is successively oriented and themagnetic field applied, the interface could be adapted to operate in acontinuous navigation or automatic mode in which the field isautomatically applied in the direction of arrow 350.

The interface also includes an advancer control pane 316. The advancercontrol pane 316 displays the length of extension of the medical devicebeing navigated. The pane 316 has three buttons: a reset zero button490, a zoom in button 492, and a zoom out button 494. The pane 316 alsohas three user settable flags 496, 498 and 500, and one system settableflag 502. The user can use the reset zero button 490 to reset thecurrent extension of the medical device as the zero position. The usercan advance and retract the medical device using the zoom in and zoomout buttons 494 and 496. The extension of the medical device from itszero position is displayed as a colored bar on the scale 504. The usercan set three flags to mark desired locations by operating the virtualbuttons 496, 498, and 500. Operating any one of the buttons causes thecorresponding flag to appear on the scale 504, and allows the user toname the flag for future reference. In modes where the systemautomatically calculates the applied magnetic field and extension toreach a particular target, the system displays the path of the device adashed line, the required field as a green arrow, and the requiredextension by positioning the system flag 502 on the scale 504. This aidsthe user in extending or retracting the medical device to the properposition to reach the target.

The interface also includes an information block 318, displaying theversion of the software, and including a pick window 506 to allow theuser to select the particular type of device being navigated. Theproperties of the device are then used in calculating and displaying theconfiguration of the device to reach a selected point, and determiningthe required magnetic field and device extension to reach the desiredpoint.

Operation

In operation the user can visualize the current direction of the devicerepresented by arrow 352 and the desired new direction for the devicerepresented by arrow 350, on the 3-D pane 306 or on the x-ray images onmonitors 86, 88, and 90. The user can selected the orientation of thearrow 350 in a number of ways using panes 308, or 310, or 314, using themenus 328 and 330 on the tool bars 304, or simply selecting a point inthe three dimensional display, and allowing the system to calculate thefield and direction to reach a selected point 16. See FIG. 18.

The user can select the orientation of arrow 350 (representing themagnetic field to apply) in a variety of ways. On pane 308 the userclicks on buttons 368 and 370, 374 and 376, and 380 and 382, to move thearrow 350 in each of the coronal or frontal plane, the median orsaggital plane, and the horizontal or transverse plane to move thearrow. Alternatively, the user can select the orientation of arrow 350by using the pane 312. The user selects a point on the menu table 408 bypointing and clicking with the cursor to set the orientation of arrow350 to the orientation associated with point selected. Alternatively,the user can select the orientation of arrow 350 using the pane 312. Theuser can select a stored orientation by selecting a category on menu428, and a direction on menu 430. The user can select a user-storeddirection by selecting a direction vector from the menu table 436. Theuser can select a previously used direction by using the buttons 432 and434 to recall one of the last previously used direction. Finally, theuser can select an orientation by picking a point on a screen 450 inpane 314.

Once the direction of the arrow 350 is selected, the navigation systemcan be operated by operating the apply button 482. This can operate amagnetic navigation system to apply a field in the direction 350, or itcan operate a magnetic navigation system to apply a field to cause themedical device to align in the direction 350, either by using feedbackof the catheter position or by calculating or using a look-up table toaccount for the properties of medical device.

A third embodiment of an interface is illustrated in FIGS. 20-26. Theinterface is adapted for controlling a magnetic navigation system thatapplies a magnetic field in a selected direction to an operating regionin a subject to magnetically orient a medical device in the operatingregion. The interface comprises a display 602 on which at least oneimage of the operating region is displayed, and in this preferredembodiment the display has panes 604 and 606 for displaying images ofthe operating region from two different planes, to facilitateidentifying points in three dimensional space in the operating region.The interface further comprises an input device, such as a mouse (notshown) for identifying points in the operating region on the at leastone image on the display, for example by moving a cursor or otherindicator over the display, and “clicking” on the selected point. Ofcourse the interface could be any other device for identifying points onthe display, including joysticks, touch screen displays, light pens,etc. By identifying a point on the image on each of the panes 604 and606, a user can uniquely identify a point in three-dimensions in theoperating region.

The interface includes a processor that, after the user selects a pointin the operating region, determining an application point in theoperating region which is on a predetermined branched path through thesubject's vasculature and which is closest to the identified point. Theinterface then determines (e.g., by calculation or use of a referencetable) the direction that is tangent to the predetermined branched pathat the application point. As shown in FIGS. 24 and 25, this directioncan be displayed by indicators 608, which may be color coded todistinguish them from indicators 609 of the previously applieddirection. The interface, preferably through the processor, then causesthe magnetic navigation system to apply a magnetic field at theapplication point, in a direction tangent to the predetermined path atthe application point.

As shown in FIGS. 22 and 23, the predetermined branched path can bemanually determined prior to beginning the procedure. A user can use theinterface to identify a plurality of points on the vasculature in theoperating region, uniquely identifying each point in three dimensionalspace by identifying it on the two panes 604 and 606, using the inputdevice (see FIG. 22). After points have been identified along thevasculature, the processor can automatically connect the points to formthe predetermined branched path by connecting each point with its nextnearest neighbors (see FIG. 23). The processor then can overlay orsuperimpose the predetermined branched path over the images of theoperating region on the panes 604 and 606, so that the user can verifythe accuracy of the branched path, and make adjustments if necessary. Asshown in FIG. 26, each branch can be displayed in a different color tohelp the user visualize the operating region. This is particularlyhelpful when viewing the operating region in two planes on the panes 604and 606. Where the vasculature curves, or is branched, the points mustbe identified fairly closely together, while wherein the vasculature instraight and unbranched, the user does not have to identify as manypoints. Alternatively, the predetermined branched path can be determinedthrough image processing, which can be assisted by the injection ofcontrast medium, if necessary.

Thus the processor creates the predetermined branched path through thevasculature in an operating region in a subject's vasculature, byaccepting the identification of a plurality of points on the subject'svasculature on at least one image of the operating region; andconnecting each point with its nearest neighboring point to form thebranched path through the vasculature.

The interface thus can be used to operate a magnetic navigation systemto apply a magnetic field in a selected direction in an operating regionin a subject, to magnetically orient a medical device in the operatingregion. The user first identifies a plurality of points along thesubject's vasculature in an image of the operating region in thesubject. The user then connecting each point to the closest adjacentpoint to create a network of navigable paths through the subject'svasculature. This can be done manually, but is preferably doneautomatically by a computer processor. The user then identifies a pointwhere on the image of the operating region, where the user wants tonavigate. The computer processor can then determine an application pointthat is on the previously determined network of navigable paths, closestto the selected point. The computer processor also determines thedirection tangent to the network of navigable paths at the applicationpoint. The interface then causes the magnetic navigation system to applymagnetic field at the application point in a direction tangent to thenavigable path at the application point.

The interface accepts the identification of a selected point on an imageof the operating region, determines an application point on apredetermined navigable path through the subject's vasculature in theoperating region corresponding that is closest to the selected point;and applies a magnetic field at the application point in a directiontangent to the navigable path at the application point. A magneticnavigation system incorporating the interface may have one or morestationary electromagnetic coils, or one or more movable electromagnetsand/or permanent magnets. The interface selectively powers thestationary electromagnets, selectively powers and moves the moveableelectromagnets, or selectively moves the permanent magnets to apply theappropriate magnetic field at the operating point in the selecteddirection.

Another control of the interface of the third embodiment is illustratedin FIG. 27. This control operates a magnetic navigation system thatapplies a magnetic field in a selected direction to an operating regionin a subject to magnetically orient a medical device in the operatingregion. The interface facilitates the specification of the direction ofthe magnetic field to be applied by the magnetic navigation system, andincludes a display pane 610 on which a representation 612 of the currentorientation of the medical device (or the currently applied magneticfield) is displayed. In this preferred embodiment the representation 612is a dot 614 at the center of a circular grid 616 comprising a pluralityof concentric circles 618 representing angular deflections from the axisof the medical device. The display pane 610 also includes a selector 620for selecting one of a plurality of predetermined patterns of neworientations. The interface includes a input device for selecting one ofthe plurality of patterns of new orientations. This input device may bea mouse and/or a keyboard for operating the selector. Of course, someother input device, such as a joystick, touch screen, etc. could be usedfor selecting a pattern,

The selector 620 includes a pick box 622 for selecting the type ofpattern. In this preferred embodiment there are preferably at least twotypes of patterns, a circular pattern generally concentric about thecurrent position of the medical device, and a spiral pattern originatingat the current position of the medical device. The selector preferablyalso includes a pick box 624 for selecting the number of new positionsin the pattern. The selector preferably also includes a pick box 626 forselecting the angular displacement of the pattern from the currentposition. The selector may also include a pick box 628 for selecting thedelay between movement among the positions in the pattern. Lastly, theselector 620 can include a previous position virtual button 630, a nextposition virtual button 632, a play virtual button 634, and a stopvirtual button 636.

The user selects the type of pattern in pick box 622, the number of newpositions in the pattern in pick box 624, the angular displacement ofthe pattern in pick box 626, and if desired a delay time in pick box628. The selected pattern is displayed on the circular grid 616 as aplurality of dots 638. The user can then operate the magnetic navigationsystem by clicking on the virtual buttons 630, 632, 634, and 636.Operating button 630 causes the interface to operate the magneticnavigation system to the previous position in the pattern. Operatingvirtual button 632 causes the interface to operate the magneticnavigation system to the next position in the pattern. Operating thevirtual button 634 causes the interface to operate the magneticnavigation system to successively move to each position in the pattern.Operating the virtual button 634 stops automatic operation of theinterface.

The colors of the representations of the new positions 638 in thepattern preferably indicate the status of each position. For example, asshown in FIG. 27B, the dots 638 b-638 h are a first color (e.g. lightgrey), indicating that the medical device has not yet been operated tothose positions. The dot 638 a is a second color (e.g. yellow),indicating it is the current position of the medical device. As shown inFIG. 27C the dots 638 b-638 h are in a first color, the dot 638 a is asecond color and a dot of a third color (e.g. green) indicating themovement of the field appears behind dot 638A. As shown in FIG. 27D, dot638 a is a fourth color (e.g. dark grey) indicating that the medicaldevice has already been navigated to the position, the dot 638 h is notthe second color, indicating it is the current position of the medicaldevice, and dots 638 b-638 g are the first color, indicating that themedical device still has not been navigated to these positions.

This pattern navigation, and automated pattern navigation, make it easyto navigate the medical device for selected procedures. For example inmapping procedures, wherein it is desirable to move a mapping catheterto trace an electrical signal, automated movement in a circular orspiral or other pattern facilitates the mapping procedure. Similarly, inablation procedures, where the user needs to move the tip of an ablationcatheter to form a closed loop of ablation, automated movement in acircular or other pattern facilitates the ablation procedure.

In operation the user can use the interface to operate a magneticnavigation system to apply a magnetic field in a selected direction inan operating region in a subject, to magnetically orient a medicaldevice in the operating region. The user selects one of a plurality ofpredetermined patterns of new positions for the medical device using theselector 320 and an input/output device, such as a mouse. The user thensimply manually operates the magnetic navigation system to successivelyorient the medical device in each new position of the pattern byoperating virtual button 632 or initiate the system automatically movingfrom position to position after the predetermined delay by operatingvirtual button 634.

A magnetic navigation system incorporating the interface may have one ormore stationary electromagnetic coils, or one or more movableelectromagnets and/or permanent magnets. The interface selectivelypowers the stationary electromagnets, selectively powers and moves themoveable electromagnets, or selectively moves the permanent magnets toapply the appropriate magnetic field at the operating point in theselected direction.

Another control of the interface of the third embodiment is illustratedin FIGS. 28 and 29. The control operates a magnetic navigation systemthat applies a magnetic field in a selected direction to an operatingregion in a subject to magnetically orient a medical device in theoperating region. The control facilitates the specification for thedirection in which to orient the medical device/apply a magnetic field.

The control comprises a display pane 650 including an indicator 652 forindicating the desired direction of the medical device and/or appliedmagnet field on a display. This indicator may be an arrow or otherelement capable of indicating a three-dimensional direction on atwo-dimensional display. The display pane 650 includes at least firstand second active areas 654 and 656 for separately controlling theindicator 652. An input device for controls a cursor or other indicatoron the display pane to click and drag within one of the two activeareas, to change the orientation of the indicator 652. Clicking anddragging in the first active area 654 rotates the indicator 652 about anaxis perpendicular to the plane of the display, and clicking anddragging in the second active area 656 flattens in the indicator intothe plane of the display, and rotates it about an axis perpendicular tothe plane of the display. The input device is preferably a mouse, butcould also be a joystick, space ball, touch screen or other device.

The indicator 652 is preferably surrounded by a closed shape, andwherein the first active area 654 outside the closed shape, and whereinthe second active area 656 is inside the closed shape. In the preferredembodiment the closed shape is a circle 658 which bounds the maximumextension of the indicator 652. The circle preferably has a plurality ofindicia around its circumference, and preferably twelve equally spacedindicia oriented like a clock face, for convenient reference by theusers.

In a preferred implementation, there are preferably multiple panesshowing the orientation of the indicator 652 from differentperspectives. As shown in FIGS. 28 and 29, panes 660, 662, and 664 canbe provided to provide an image of the indicator from three mutuallyperpendicular perspectives. Each of the panes allows for rotation of theindicator about an axis perpendicular to its particular plane. This, asdescribed above, allows the user to adjust the orientation of theindicator 652.

The indicator 652 in pane 660 is surrounded by a circular frame 666,defining a first active area 668 outside the frame, and a second activearea 670 inside the frame. Clicking and dragging in first active area668 causes the indicator to rotate about an axis perpendicular to theplane of pane 652, while clicking and dragging in second active area 670causes the indicator to drop into the plane of the pane 660, and rotatein that plane about an axis perpendicular to the plane of the pane 660.

The indicator 652 in pane 662 is surrounded by a circular frame 672,defining a first active area 674 outside the frame, and a second activearea 676 inside the frame. Clicking and dragging in first active area674 causes the indicator to rotate about an axis perpendicular to theplane of pane 652, while clicking and dragging in second active area 676causes the indicator to drop into the plane of the pane 662, and rotatein that plane about an axis perpendicular to the plane of the pane 662.

The indicator 652 in pane 664 is surrounded by a circular frame 678,defining a first active area 680 outside the frame, and a second activearea 682 inside the frame. Clicking and dragging in first active area680 causes the indicator to rotate about an axis perpendicular to theplane of pane 664, while clicking and dragging in second active area 680causes the indicator to drop into the plane of the pane 664, and rotatein that plane about an axis perpendicular to the plane of the pane 664.

In operation the interface is used to control a magnetic navigationsystem to apply a magnetic field in a selected direction in an operatingregion in a subject to magnetically orient a medical device in theoperating region. The user selects the direction in which to apply amagnetic field by clicking and dragging on one of first and secondactive areas of a display to rotate an indicator indicating the desireddirection. Clicking and dragging on the first active area rotating theindicator about an axis perpendicular to the plane of the display, andclicking and dragging on the second active area collapsing the indicatorinto the plane of the display, and rotating it about an axisperpendicular to the plane of the display. The user then operates theinterface to cause the interface to apply a magnetic field to theoperating region in the direction indicated by the indicator.

As shown in FIG. 3 this last control mode of navigation can be appliedto specifying the field on the other panes. In this plane mode, a clocklike circle is superposed over the indicator of the desired newdirection. The indicator can be moved about an axis perpendicular to theclock face by clicking and dragging outside the clock face, or it can bemoved in the plane of the clock face, also about an axis perpendicularto the clock face, by clicking and dragging inside the clock face.

A fourth embodiment of an interface in accordance with the principles ofthis invention is illustrated in FIGS. 32-48.

One possible embodiment of a main menu for display on the monitors 72and 80 is indicated generally as 700 in FIG. 32. The menu 700 hasvirtual buttons which the user can select and click using either themouse 74 or 84, or using the joystick 76 or 86. In this preferredembodiment there are buttons 702 for operating the interface in thebasic mode, button 704 for using the interface in an EP procedure;button 106 for using the interface in and IC procedure, and a button 708for using the interface during a CRT procedure. The menu preferably alsohas buttons 710 for opening a previous procedure, a button 112 forimporting a previous procedure, and a button 714 for exporting aprevious procedure, and a button 716 for shutting down the system.

To enter the basic navigation mode of the interface, a user points thecursor at the “basic” button 702, and clicking with the mouse orjoystick. This action causes the basic navigation display, indicatedgenerally as 800 in FIG. 33, to be displayed on the monitors. As shownin FIG. 33, the display 800 preferably includes a menu bar 802, which ispreferably common to all screens, and which in this preferred embodimenthas categories “Procedure”, “Navigation”, “Fluro”, “View”, “Volume”,“Settings”, and “Help”. The display 800 preferably also includes a menubar 804, which is preferably common to all screens, and which in thispreferred embodiment has a step selector box 806, for selecting theincrement size for changes in the direction of the applied magneticfield, and a magnetic navigation selector box 808, for selecting betweena “Manual Apply” mode in which the magnetic field must be manuallyapplied after the magnetic field direction is selected, and an“Automatic” mode in which a magnetic field is automatically applied inthe selected direction without further action by the user. There arealso format selection buttons 814 and 816, for selecting the format ofthe display. In this preferred embodiment there are two formats, thefirst, selectable with button 814 has two display panes surrounded bycontrol panes at the left side and the top (for example as shown in FIG.33), and the second, selectable with button 816, has four display panessurrounded by control panes at the left and right sides (for example asshown in FIG. 34A). The user can select the type of display pane orcontrol pane to display by right clicking in a particular area todisplay a menu, and selecting the particular type of display panedesired.

An x-ray button 818 can be operated to drop down a menu to select anx-ray image to display in an active x-ray image pane on the display. Anx-ray transfer button 820 can be operated to transfer new x-ray imagesfrom an x-ray imaging system to the interface, so that the new image canbe selected when x-ray button 818 is operated.

A store point button 822 can be operated to store the current locationof the distal end of the medical device being navigated with the userinterface, using the Points & Constellation Pane as described below.Similarly a store vector button 824 can be operated to store the currentdirection of the device (or in this preferred embodiment the currentmagnetic field direction applied to the device by the magneticnavigation system. There is a vector window 826 for displaying thestatus of the applied magnetic field. As a newly selected magnetic fieldis applied, an indicator in the window 826 increases from 0% to 100%until the selected field is applied.

The display 800 also includes various user-selected panes for displayingthe current orientation of the medical device and for specifying a newdesired location or orientation of the medical device. As shown in FIG.33, these panes can include a 2D field specification pane 828, anavigations pane 830, a presets pane 832, and a bulls eye pane 834. Asdescribed above, each of these panes can be replaced by right clickingon the pane to display a list of available alternative panes which canbe selected. In this preferred embodiment, these panes can include abull's eye pane, a points & constellations pane, a catheter adjustmentpane, an alignment pane, and 2-D anatomical navigation pane. Of coursethe display 800 could include additional panes or fewer panes ordifferent panes, including the various panes disclosed in the variousother embodiments of this invention.

As also shown in FIG. 33, these panes also include two large panes,which can be an object navigation pane 836 and a field specificationpane 838, which displays one of the subpanes of the 2-D fieldspecification pane 828. As described above, each of these panes can bereplaced by right clicking on the pane to display a list of availablealternative panes which can be selected. In this preferred embodiment,these panes can include an object navigation pane, a 3D sync pane, apreoperative image pane, and an x-ray pane. Of course the display 800could include additional panes or fewer panes or different panes,including the various panes disclosed in the various other embodimentsof this invention.

The “Navigations” pane 830 includes a list 840 for entering a name ordescription of a stored direction; a store button 842 for storing thecurrent direction (similar to button 824, described above); an editbutton 844 for editing the name or description of a previously storednavigation; a delete button 846 for deleting a previously storednavigation; and a reference box 848 with a set button 850 and a removebutton 852 for setting the color and id for a stored direction.

The “Presets” pane 832 allows the user to quickly select directions fromone or more sets of standard directions. The “Presets” pane 832 has alist subpane 854, with an icon column 856, a list column 858 for thename of one or more sets of standard or preset directions. The “Presets”pane 832 also includes a directions subpane 860, which displays thedirections that are part of a selected set on the list 858. Thedirections subpane 860 includes an ID column 862 and a descriptioncolumn 864. The ID column 862 contains a reference and a color code foreach direction in the set, and the description column 864 contains aname or description of each direction in the set.

The Bull's Eye pane 834 comprises a generally circular display screen868, and a plurality of buttons, including a set center button 870, ashow/hide button 872, switch orientation button 874, and scale menu 876.In the preferred embodiment, the set center button 870 centers thebull's eye grid on the current magnetic field direction, so that it isconcentric with the indicator of the current magnetic field direction.In other embodiments of the user interface for use with other navigationsystems, this button could center the bull's eye screen on the currentdirection of the medical device instead of the current magnetic fielddirection. The show/hide button 872 toggles between a show mode in whichthe bulls eye grid is shown in the display panes in the interface, and ahide mode in which the bulls eye display is not shown in the displaypanes.

The switch orientation button 874 changes the orientation of the bull'seye grid between the standard proximal to distal view and a distal toproximal view 876. The display screen 868 preferably has indicators forindicating the orientation of the display screen and the correspondingbull's eye grid. For example, as shown in FIG. 33, an indicator 838 a,which may be color coded (e.g. blue) is shown at the top of the circulardisplay screen 838, and a corresponding indicator is shown on the bullseye grid in the display panes, in the superior (up direction). A secondindicator 838 b, which also may be color coded (e.g. red) is shown atthe left side of the circular display screen 838, and a correspondingindicator is shown on the bull's eye grid in the display panes. Theindicators 838 a and 838 b help the user to understand the relationshipbetween the display screen 838 and the bull's eye grid in the userinterface. The switch orientation button 838 allows the user to switchthe direction of the direction of the display screen 838 and thecorresponding bull's eye grid in the display panes, from a perspectivelooking toward the distal end of the device, and the thus the indicator838 b is on the left side of the display screen 838 and thecorresponding bull's eye grid, and a perspective looking toward theproximal end of the device, and thus the indicator 838 b is on the rightside of the display screen, and the corresponding bull's eye grid,reflecting the fact that the left side looking distally is the rightside looking proximally.

The scale orientation menu 876 allows the user to set the scale of thedisplay 838 and the bull's eye grid. In this preferred embodiment, theuser can right click on the menu 876, and select from one of severalpreset scales, preferably including 15°, 30°, 45°, 60° and 90°. TheBull's eye navigation pane can otherwise be used like the bull's eyenavigation pane 610 shown in FIG. 27, and described above. Thus, theBull's Eye pane 834 preferably also includes a pattern navigation box880 for selecting a pattern of points and for manually or automaticallynavigating to the points in the selected pattern. This patternnavigation box 880 includes a box 882 for activating pattern navigation,a selection box 884 for selecting the shape of the pattern, a pointsselection box 886 for selecting the number of points in the pattern, arange selection box 888 for selecting the angular range of the patternfrom the center point; and a pause selection box 890 for selecting theduration of the pause between each navigation to a point in theautomated pattern navigation mode. A control bar 892 has a first button894 for returning to the first point in the pattern, a last button 896for moving to the last point in the pattern, an advance button 898 foradvancing to the next point in the pattern, and a autoadvance button 900for automatically advancing through successive points, and a stop button902 for stopping the automatic advancement.

The interface preferably displays a visual indicator of the desiredorientation for the distal end of the medical device, such as the arrow150, whose shaft is aligned with the desired orientation, with a largeconical head pointing in the desired direction. The arrow 150 ispreferably a distinctive color such as green. The interface preferablyalso displays a visual indicator of the current orientation of thedistal end of the medical device, such as arrow 152, whose shaft isaligned with the current orientation of the distal end of the medicaldevice, with a larger conical head pointing in the desired direction.The arrow 152 is preferably a distinctive color such as yellow.

A localization system can be provided for determining the currentposition and orientation of the distal end of the medical device. Animage representative of the distal end of the medical device can then begenerated and displayed. There are numerous method for localizing thedistal end of the medical device, for example transmitting magnetic orrf signals between the medical device and one or more referencelocations, x-ray image processing, ultrasound localization, or electricpotential localization.

In this preferred embodiment, the interface is adapted for use with amagnetic navigation system that operates by generating a magnetic fieldof selected direction in the operating region, which causes amagnetically responsive element associated with the distal end of themedical device to generally align with the applied magnetic field.Because of the physical properties of the medical device, limitations inthe strength of the applied field, and the conditions in the proceduresite, the distal end of the medical device may not align precisely withthe applied magnetic field. While the difference between the appliedmagnetic field and the actual direction of the distal end of the medicaldevice can be accounted for through modeling or a look-up table, in thispreferred embodiment the arrow 150 representing the desired orientationmay represent the desired direction of the applied magnetic field,rather than the desired direction of the medical device itself.Similarly, the arrow 152 representing the current orientation mayrepresent the direction of the magnetic field currently being applied,rather than the actual direction of the device itself. However, thedifferences between the actual direction of the medical device and theapplied magnetic field can be characterized by equation or anempirically determined look-up table, or localization of the device canbe provided so that even when used with a magnetic navigation system,the arrow 150 represents the actual desired orientation of the medicaldevice, and arrow 152 represents the actual current direction.

Other panes facilitate the visualization and selection of directions.One such pane to aid the user in selecting the desired orientation forthe medical device is the 2-D anatomical pane 828, which allows the userto select the desired direction (as indicated by the arrow 150) byadjusting the direction of the arrow 150 in one or more planes throughthe operating region in the subject. As shown in FIG. 33, the pane 828allows the user to change the direction of the arrow 150 in at least oneplane, and preferably at least two planes and more preferably at leastthe planes. These planes are preferably, but not necessarily, mutuallyperpendicular. While adjustment in two planes is sufficient to specifyany direction, providing adjustment in three planes makes it easier fora user to select the desired direction for the arrow 150. In this fourthpreferred embodiment, the arrow 150 can be rotated in the coronal orfrontal plane (i.e., about an anterior-posterior axis), the median orsaggital plane (i.e., about a horizontal axis), and the horizontal ortransverse plane (i.e., about a longitudinal axis).

As shown in FIG. 33 the pane 828 can have three subpanes 910, 912 and914, corresponding to the three planes of rotation. Subpane 910 containsa graphic depiction of the coronal or frontal plane (i.e., a caricatureimage of a subject's body in the coronal or frontal plane), with anindicator 916 that indicates the orientation of the arrow in the coronalor frontal plane, and virtual buttons 918 and 920 for moving theindicator 916 (and thus the arrow 150) clockwise or counterclockwise inthe coronal or frontal plane abut the anterior-posterior axis. In thisfourth preferred embodiment, indicator 916 is actually a projection ofthe arrow 150 in the plane, and thus the length of the indicator 916 isindicative of the orientation. The virtual buttons 918 and 920 can beoperated with a cursor for example with the mouse 74 or 82 or thekeyboard 80, to point and click the button and move the indicator 916and thus the arrow 150, in the desired direction.

Subpane 912 contains a graphic depiction of the median or saggital plane(i.e., a caricature image of a subject's body in the median or saggitalplane), with an indicator 922 indicating the direction of the arrow 150in the median or saggital plane, and virtual buttons 924 and 926 formoving the indicator 922 (and thus the arrow 150) clockwise orcounterclockwise in the coronal or frontal plane. In this firstpreferred embodiment, indicator 922 is actually a projection of thearrow 150 in the plane, and thus the length of the indicator 922 isindicative of the orientation. The virtual buttons 924 and 926 can beoperated with a cursor for example with the mouse 74 or 82 or thekeyboard 80, to point and click and move the indicator 922 and thus thearrow 150, in the desired direction.

The pane 914 contains a graphic depiction of the horizontal ortransverse plane (i.e., a caricature image of a subject's body in thehorizontal or transverse plane), with an indicator 928 indicating thedirection of the arrow in the horizontal or transverse plane, andvirtual buttons 930 and 932 for moving the indicator 178 (and thus thearrow 150) clockwise or counterclockwise in the horizontal or transverseplane. The virtual buttons 930 and 932 can be operated with a cursor forexample with the mouse 58 or 66 or the keyboard 64, to point and clickto move the indicator 928, and thus the arrow 150 in the desireddirection.

As described above, the menu bar 804 contains a step selector box 806 toselect the increment of change in direction upon operating the buttons918 and 920, 924 and 926, and 930 and 932. The user can preferablyselect the incremental change from several preset increments: 1 degree,2 degrees, 3 degrees, 5 degrees, 10 degrees with a cursor for examplewith the mouse 74 or 82 or the keyboard 80, to point and click to selectthe desired increment.

The object navigation pane 836 has a representation 940 of athree-dimensional object. This three dimensional object is preferably asphere, but it could be some other shape such as an ellipse or a cube.There are preferably indicators on the surface of the three dimensionalobject to indicate the corresponding directions in the operating regionin the subject. In this preferred embodiment, these indicators includecones 942 for identifying the directions of major anatomical axes in thesubject. As shown in FIG. 33, there are a pair of cones 942A identifyingthe superior-inferior axis, a pair of cones 942B identifying theanterior-posterior axis, and a pair of cones 942C identifying theleft-lateral right-lateral axis. Each of the pairs of cones 942A, 942B,and 942C, are preferably displayed in a unique color. There arepreferably a plurality of latitude lines and longitude lines on thesurface of the three-dimensional object. In this preferred embodiment,the latitude lines and longitude lines for each axis are colorcoordinated with the color of the cones, and preferably can beselectively displayed or hidden.

The indicators can also include direction indicators 944 for identifyingother selected directions in the operating region in the subject on thesurface of the three dimensional object. As shown in FIG. 33, theseindicators 944 can include an indicator 944A indicating the RAOdirection and an indicator 944B indicating the LAO direction. The conepairs 942A, 942B, and 942C, and the indicators 944A and 944B, indicatethe preset cardinal directions as set forth on the Presets pane 832.

The indicators can also include a representation of at least a portionof the subject, such as representation 946. This representation can bean idealized representation of a subject. Alternatively, thisrepresentation could be created from actual image data of the subject.Of course the representation could also be a representation of aninternal body structure. This representation could either be anidealized representation of the internal body structure, or it could bea representation created with imaging data from the subject. Each pointon the surface of the representation 940 of the object corresponds to adirection in the operating region. The user clicks on a location on thesurface of the representation 940 of the object to identify the desireddirection of orientation in the operating region. The indicators 942 and944, and the representation 946 help the user identify the point on thesurface of the representation of the object 940 that corresponds to thedesired direction.

The object navigation pane 836 has a rotation button 948, aspherical/hemispherical button 950, a 3D sync button 952, and a viewselection window 954. The rotation button 948 toggles between a rotationmode in which the cursor can be manipulated by a control device such asmouse, joystick, or keyboard to grab and rotate the representation 940of the object, and a selection mode in which the cursor can bemanipulated by a control device such as a mouse, joystick, or keyboard,to select a point of the surface of the representation 940 of theobject.

The spherical/hemispherical button 950 toggles between an external viewof the representation 940 of the object, and an internal view of therepresentation of the object, which is preferably taken from a verticalplane through the center of rotation of the object.

The 3D sync button 952 preferably synchronizes rotation of a 3Drepresentation of the operating region with the rotation of therepresentation 940 of the object in the object navigation pane 838.

A floating “Fluro” tool bar 956 can be made to appear on the interfacedisplay, and has a contrast/brightness button 958, a pan button 960, anda vector show/hide button 962. The contrast/brightness button 958displays a menus allowing the user to adjust the contrast and thebrightness of the fluoro display pane (which is not shown if FIG. 33).The pan button 960 allows the user to pan over the x-ray image displayedin the fluoro display pane. The vector show/hide button 962 can beoperated by the user to selectively show or hide the direction vectors,e.g. the current direction vector 150 and the desired new directionvector 152.

A floating “Mode” window 964 can also be made to appear on the interfacedisplay, and has a vector mode button 966, an x-ray mode button 968, atarget mode button 968, an a vessel navigation mode button 972. Thevector mode button 966 causes the active display panes to enter thevector mode in which the directions are displayed and specified on thepanes. The x-ray mode button 968 permits field adjustments to be made inthe plane of the selected reference X-ray window. The target mode button968 causes the active display panes to enter the target mode, wheretarget points are identified rather than navigation directions. Thevessel navigation button can be used to set the local tangent at alocation along a vessel constellation (defined by an automatic splinecurve fitting to a group of fluoro-localized points along a vessel) bydouble-clicking at that location on the vessel constellation in the 3Dwindow.

A display from the EP mode of the fourth preferred embodiment of a userinterface is indicated as 1000 in FIG. 34. The display is reached byclicking on button 704 on the menu 700. The display 1000 comprises a 2Dfield specification pane 1002, similar to pane 828 described above. Thedisplay 1000 also comprises a “Points & Constellation” pane 1004, a“Presets” pane 1006 similar to “Presets” pane 832 described above, an“Alignment” pane 1008, and a “Catheter Adjustment Tools” pane 1110. Thedisplay 1000 also includes a 3-D view pane 1012, an anatomical modelpane 1014, an x-ray display pane 1016, and an object navigation pane1018 similar to the object navigation pane 836.

Using the various navigation tools of the interface, the user cannavigate the distal end of a medical device to selected points in theoperating region, and assemble these points in groups or constellations.The user can use the “Points & Constellations” pane 1004 to select howthese points are displayed in the various panes of the user interface.As shown in FIG. 34, the “Points & Constellations” pane 1004 has asubpane 1020 with a column 1022 for selecting how the point orconstellation is displayed on the panels of the display 1000; a column1024 for indicating a color code for displaying the point or group ofpoints, and a code for indicating from which panes the points werecreated, a column 1026 for displaying a name of the point or group ofpoints; a column 1028 for selecting whether to display the point orgroup of points on an x-ray display pane 1016, and a column 1030 forselecting whether to display the point or group of points on the 3Ddisplay 1012. When a particular point or group of points on the subpane1020 is selected, the member points are displayed in subpane 1032, whichhas a column 1034 for an ID of the point and a column 1036 for a name ofthe point.

The column 1022 allows the user to select how the points in a group aredisplayed in the 3-D pane 1012. The user can select (for example byright clicking with the mouse or some similar command) to display thepoints as a group. The user can also select to display a group of pointsas a computer generated spline (a closed loop that best fits the pointor points selected. (See FIG. 35). The user can also select to display agroup of points as a computer generated planar disk, such as disks 1038in pane 1012, which are particularly well suited for displaying ostiumor openings in a surface that have been identified by a plurality ofpoints. The interface generates a closed loop of active points 1036 froma set of selected points, and generates a disk of active pointssurrounding the loop in the same general plane. (See FIG. 36). The usercan easily select a point on the loop 1036 or on the disk 1038, and theinterface allows the user to make the medical device to point toward theselected point. The user can also select to display the points as asurface patch 1039, comprising a surface between the points in thegroup. (See FIG. 37).

The user can also cause the points to be displayed in the x-ray imagepane 1016, by checking the appropriate box in column 1028 of the subpane1032 As shown in FIGS. 34A and 34B, the groups are indicated as splinessuperposed over the x-ray image in the pane 1016, in the color indicatedin column 1024 for the group to which each point belongs.

The pane 1012 has a pan button 1040, a 3D grid show/hide button 1042, apoint selection button 1044, a direction selection button 1046, a bullseye button 1048, a first user defined view_button 1050, a second userdefined view button 1052, a third user defined view button 1054, and aview selection window 1056. The pan button 1040 allows the user toselect a mode in which the movement of the cursor, for example with themouse, joystick, or keyboard, allows the user to pan across the view inthe pane 1012. The point selection button 1044 allows the user to selecta point in the pane 1012. The direction selection button 1046 allows theuser to select a direction in the pane 1012. The bulls eye button 1048causes the bull's eye grid to be displayed on the pane 1012. The first,second, and third user defined view buttons 1050, 1052, and 1054 allowsthe user to save up to three separate user defined views, so that theuser can quickly and easily return to those views. The view selectionwindow 1056 allows the user to select one of several standard views inthe pane 1012.

The object navigation pane 1018 is similar to the object navigation pane836 on display 800, and includes a representation 1058 of a threedimensional object. As described above with respect to object 940, thethree dimensional object depicted by representation 1058 is preferably asphere, but it could be some other shape such as an ellipse or a cube.There are preferably indicators on the surface of the representation1058 of the three dimensional object to indicate the correspondingdirections in the operating region in the subject. In this preferredembodiment, these indicators include cones 1060 for identifying thedirections of major anatomical axes in the subject. As shown in FIGS.34A and 34B, there are a pair of cones 1060A identifying thesuperior-inferior axis, a pair of cones 1060B identifying theanterior-posterior axis, and a pair of cones 1060C identifying theleft-lateral right-lateral axis. Each of the pairs of cones 1060A,1060B, and 1060C, are preferably displayed in a unique color. There arepreferably a plurality of latitude lines and longitude lines on thesurface of the three-dimensional shape. In this preferred embodiment,the latitude lines and longitude lines for each axis are colorcoordinated with the color of the cones, and preferably can beselectively displayed or hidden.

The indicators can also include direction indicators 1062 foridentifying other selected directions in the operating region in thesubject on the surface of the three dimensional object. As shown inFIGS. 34A and 34B, these indicators 1062 can include an indicators forthe various preset directions for the selected set of directions in“Presets” pane 1006. The “Presets” pane 1006 is similar to the “Presets”pane 832 described above. The cone pairs 1060A, 1060B, and 1060C, andthe indicators 1062 help the user select the appropriate point on thesurface of the representation 1058 of the object to properly identifythe desired corresponding direction in the operating region in thesubject.

The indicators can also include a representation of at least a portionof the subject, such as representation 1064. This representation 1064can be an idealized representation of the internal body structures inthe operating region in the subject. Alternatively, this representation1064 could be created from actual image data of the operating region inthe subject. Of course the representation could also be a representationof an external body structure, as described above with respect to pane836 of display 800.

The object navigation pane 1018 is particularly adapted for directionbased navigation, rather than point based navigation, although it couldbe used in point based navigation. Each point on the surface of therepresentation 1058 of the object corresponds to a direction in theoperating region. The user simply manipulates a cursor to the desiredposition and clicks on the location on the surface of the representationof the object to identify the desired direction in the operating regionin the subject. The indicators 1060 and 1062, and the representation1064 help the user identify the point on the surface of therepresentation 1058 of the object that corresponds to the desireddirection.

Other directional markers are preferably provided on the surface of theobject 1058. For example, the various system-specified anduser-specified preset directions can be stored by the interface. The“Presets” pane 1006 is similar to Presets pane 832, and correspondingparts are identified with corresponding reference numerals. The“Presets” pane has a “List” subpane 854, and a “Directions” subpane 860.The list subpane 854 has a column 856 for icons indicating the type ofset of directions, and a name column 858 for indicating the name of thedirection. For a selected set of directions on the List subpane 854, the“Directions” subpane 860 displays the various directions that comprisethe set. The “Directions” subpane 860 has an ID column 862 whichdisplays an ID code and a color code for each direction, Thesedirections are in turn displayed on the object 1058 using indicators1062 with corresponding ID codes and colors. These indicators 1062 helpthe user select a point on the surface of the object 1058 thatcorresponds to the desired direction in the operating region, includingpicking one of the preset directions.

The interface is preferably provided with a variety of presentdirections that users can employ to orient the medical device. Forexample standard directions from the center of the operating region, orfrom a particular entrance of an operating region, to common proceduredestinations for typical anatomies can be determined, and thesedirections stored, so that when a user desires to navigate to one ofthese common destinations, the user merely needs to select from amongthe standard directions. For example, as shown in FIG. 34A and 34B,standardized directions to the center of the mitral center, the 3o'clock position on the mitral valve, the six o'clock position on themitral valve, and the 12 o'clock position are all indicated by markerson the object 1058. These marks are color coordinated with the ID column862 in the “Directions” subpane 860, and each is identified with thecode appearing in the ID column 862.

The user can select a direction by pointing a cursor and clicking on thesurface at the point corresponding to the desired direction. In thispreferred embodiment which includes a magnetic navigation system, themagnetic navigation system is then applied, for example by clicking onthe apply magnetic field button 1066. Depending upon how the system isset up, the system may either apply a magnetic field in the selecteddirection, or it may apply a magnetic field that will cause the deviceto point in the desired direction. Alternatively, in the continuousmode, the magnetic navigation system can automatically act to orient thedevice in the desired direction. When used with non-magnetic navigationsystem, the selected direction can be input to the control of thenon-magnetic navigation system, which can then orient the medical devicein the desired direction.

The pane 1018 includes an orientation button 1068, afull-sphere/half-sphere button 1070, a 3D sync button 1072, and a window1074. The orientation button 1068 causes the display to enter the pivotmode where the user can uses the controls (e.g. the cursor under thecontrol of the mouse or joystick), to rotate the object 1058. The object1058 preferably pivots about its center, but it could pivot about someother point, if desired. The full-sphere/half-sphere button 1070 togglesbetween a full-sphere view in which the entire object 1058 is visible(for example, FIG. 34A), and a half-sphere view in which the front halfof the object 1058 is removed, and the rear half is visible from theinside (for example FIG. 34B). The 3D sync button 1072 synchronizes theobject navigation pane 1018 with the 3D display pane 1012, so thatrotation of the representation of the object 1058 rotates that 3ddisplay pane 1012. The window 1074 allows the user to select among oneof several standard views.

The interface can also employ a Catheter Adjustment Tool pane 1010,which has an angle adjustment box 1076, and a position adjustment box1078, and a position adjustment box 1080. As shown in FIG. 38, in theangle adjustment box 1076, the user can specify an angle of catheterover-torque, i.e., an angular adjustment beyond what is necessary tocontact the surface, to increase the contact force between the catheterand an anatomical structure in the operating region. The box 1076includes an indicator window 1082 for displaying the angle ofover-torque, and increase and decrease buttons 1084 and 1086 forincreasing and decreasing the angle of over-torque, and a slide control1088 for alternatively specifying the angle of overtorque. A button 1112can be provided to reset the over-torque to zero.

The position adjustment box 1080 includes controls for displaying andoperating a distance adjustment tool. The box is used in conjunctionwith a display of a representation 1081 of the distal end of the medicalto adjust the configuration of the representation of the distal end of amedical device to a desired configuration, and then cause the actualmedical device to conform to the configuration of the representation ofthe medical device. The representation 1081 is preferably derived from acomputational model of the actual medical device, as is discussed inU.S. patent application Ser. No. 10/448,273, filed May 29, 2003, forRemote Control of Medical Devices Using a Virtual Device Interface, U.S.patent application 20040068173 published Apr. 8, 2004, the entiredisclosure of which is incorporated herein by reference.

The box 1080 includes a button 1118 for displaying a catheter adjustmentindicator or reticle 1120 (shown in FIG. 39), in the 3D view pane (e.g.,panes 1012 in FIGS. 34A and 34 b). In this preferred embodiment thereticle 1120 comprises a series of concentric rings (hexagonal in thispreferred embodiment) with a perpendicular line extending through itscenter point, although the active area of adjustment is really anumbrella-shaped surface that includes the displayed rings. The reticle1120 is displayed at a point selected by the user. If the point selectedby the user (for example by pointing a cursor and clicking) is not on asurface, the reticle 1120 is preferably displayed centered at the distaltip of the representation of the medical device, in an orientationperpendicular to the orientation of the distal end of the medicaldevice. If the point selected by the user is on a surface, the center ofthe reticle 1120 is preferably positioned on a line normal to thesurface point at a default distance from the surface selected by theuser. This default distance may be zero to position the reticle 1120 atthe surface, or it may be a positive distance to position the reticle1120 beyond the surface, or a negative distance to position the reticleshort of the surface. In this preferred embodiment, the user can alsospecify that the reticle be displayed in the plane of the display at theselected point by using button 1116.

The position of the reticle 1120 and thus the position of therepresentation of the distal end of the medical device, which followsthe reticle, can be adjusted to achieve a desired position orconfiguration of the representation of the medical device. Once thedesired position or configuration is achieve, the navigation system canbe operated to cause the actual medical device to conform to therepresentation. Alternatively, the navigation system could automaticallyoperate to cause the actual medical device to conform to therepresentation of the medical device as changes are made to therepresentation.

The box includes an indicator 1098 which indicates the distance betweenthe reticle 1120 (and the distal end of the medical device beingcontrolled with the interface), and the starting point. Where an offsetfrom the surface is specified, this offset is automatically displayed inthe indicator 1098. Further movement of the reticle 1120 is reflected inthe indicator 1098. Buttons 1100 and 1102 allow the user to selectivelyincrease or decrease the distance between the reticle 1120 and therepresentation of distal end of the medical device) and the startingpoint. The box 1080 can also include a slide control 1104 for selectingthe distance (+ or −) of the reticle 1120 and the representation of thedistal portion of the elongate medical device and the starting point,increasing and decreasing the distance between the reticle and thestarting point, and thereby changing the configuration of therepresentation of the distal end of the medical device. This distancecan be either negative indicating that the desired destination point isinside of the starting point, or the distance can be positive indicatingthat the desired destination point is outside starting point.

As shown in FIG. 38, the box 1080 can also include a surface persistentbox 1106. When the user selects the surface persistent box 1106, theindicator 1120 remains at its current location even if the distal tip ofthe medical device moves. A store button 1108 stores the location of theindicator 1120, and the recall button 1110 recalls the location of theindicator 1120. This facilitates returning to a previously navigatedlocation.

As shown in FIG. 38, the box 1078 can have a window 1090 indicating thechange in length of the elongate medical device to reach the destinationpoint represented by the reticle. The window indicates amount of thechange, and increase and decrease indicators 1092 and 1094 indicate thedirection of change (i.e. whether the length needs to be increased ordecreased) to bring the distal end of the actual device to the pointindicated by the representation of the distal end of the medical deviceon the reticle 1120. When the free length of the medical device needs tobe increased to reach the position in the operating region correspondingto the position represented by the representation of the medical deviceon the reticle 1120, the increase indicator 1092 is activated, and whenthe free length of the medical device needs to be decreased to reach theposition in the operating region corresponding to the positionrepresented by the representation of the medical device on the reticle1120 indicator, the decrease indicator 1094 is activated. The box 1078also includes a reset button 1096 to reset the window 1090.

In operation the user navigates the distal end of the device to a pointon the surface, and clicks on the show/hide button 1118. The reticle1120 comprising a set of concentric rings is displayed centered at theselected point in the operating region. Alternatively, the user couldoperate button 1114 to display the indicator 1120 at the distal end ofthe medical device, perpendicular to the current orientation of thedistal end of the device, or the user could operate button 1116 todisplay the indicator 1120 at the distal end of the medical device, in aplane perpendicular to the plane of this pane. A line 1122 normal to thesurface at the selected current point is also displayed. The useradjusts the position of the reticle 1120 to identified the desireddestination of the distal end of the medical device in the operatingregion. The representation of the medical device, which is preferablybased upon a computational model, displays the predicted or theoreticalconfiguration of the distal end portion of the medical device. The useradjusts the position of the reticle 1120 along the normal line 112 byoperating the buttons 1100 and 1102, or operating the slide control1104. The reticle also defines an active surface, and the user can alsomove the representation of the distal end of the medical device from thecenter of the recycle to some other point on the surface of the reticleby selecting the point on the surface of the reticle (for example bypointing with a cursor and clicking). The representation of the medicaldevice updates to display the predicted or theoretical configuration ofthe medical device.

By adjusting the position of the reticle 1120 along the line 1122, andby adjusting the position of the representation of the medical devicerelative to the surface of the reticle, the user can accurately makeminor adjustments in the position and configuration of the actualdevice. Once the desired configuration of the medical device iscorrectly indicated in the display, the user can then apply theconfiguration. In this preferred embodiment employing a magneticnavigation system, the user simply click the apply field button and thenavigation system orients the actual device in the direction required toreach the target point indicated on the display. The user then advancesor retracts the device as indicated on the indicators 1090-1094, tobring the medical device to the desired destination. With othernavigation systems, the system could be operated to orient the distalend portion in the appropriate configuration, and the device advanced.In this manner the distal end of the device can be easily advanced intoa lumen, or pressed against a wall of an anatomical structure.

It is possible that the movement of the distal end of the device couldbe automated as well as the orientation of the distal end. Feedback froma localization system could also be incorporated to facilitate theautomation of the orientation and movement of the distal end of thedevice.

The display 1000 also includes an anatomical pane 1014, with arepresentation 1126 of a three-dimensional anatomical feature, obtainedfrom imaging data from the subject. Each point on the surface of therepresentation corresponds to a location in the operating region. Theanatomical pane 1014 has a pan button 1128, a show/hide objects button1130, a center point button 1132, a fit-objects-to-window button 1134, aview synchronization button 1136, a transparency selection button 1138,a Tissue Editor button 1140, and a_“Share Fluoro Points” button 1142.The pan button 1128 allows the user to use the cursor to click and dragthe display in the pane, for example with the mouse, joystick, orkeyboar to change the view. With a mouse click on this button, a toolbarof other choices for the mouse mode (such as rotation) can also bebrought up to change the mouse mode if desired. The show/hide objectsbutton 1130 allows the user to display objects such as the pointsselected on the surface of the representation 1126 of the threedimensional anatomical feature, within the anatomical window pane 1014.The center point button 1132 centers the display on the pane 1014 on aselected point. The fit-objects-to-window button 1134 rescales thedisplay scale within the window so that all objects are visiblydisplayed in the pane 1014. The view synchronization button 1136synchronizes the display perspective in the pane 1014 to that of thedisplay selected with this button, for example that of the bull's eyedisplay. The transparency selection button 1138 allows for a choice ofopaque, semi-transparent or transparent display of the three dimensionalimage displayed in the pane 1014. The Tissue Editor button 1140 bringsup a choice of colors, shades and transfer functions for the data thatmay be chosen by a user to display the three dimensional image data incustomized form. The “Share Fluoro Points” button 1142 permits displayof fluoro-localized points in the pane 1014 after a suitableregistration between preoperative image data and the X-ray or navigationsystem has been effected, in a manner similar to that described herein.

The pane 1014 also includes a window 1144. The window 1144 allows theuser to selected one of several standard views.

Once the three dimensional image data has been registered to thenavigation system, the user can manipulate the representation 1126 andselect a target by picking a point on the surface of the representation.As also shown in FIG. 34A, the user can also employ the distanceadjustment tool to advance and retract the medical device along a linenormal to the surface of a selected target point.

The alignment pane 1008 can be used to align or register the 3D image inpane 1014 with the x-ray image in pane 1016. As best shown in FIG. 49Aand 49B, the alignment pane has tabs to select between an auto(automatic) mode (FIG. 49A) and a manual mode (FIG. 49B). In theautomatic mode shown in FIG. 49A, the pane has a fluoro point box 1200,with an ID column 1202 and a name column 1204, and a preop point box1206 with an ID column 1208 and a name column 1210. In the auto mode,pane 1008 also includes an “Add Selected Points” button 1212, and“Adjust” button 1214, and a “Reset All” button 1216. The user selectspoint from the Points & Constellations pane 1004. The user picks a groupfrom column 1026, and then selects one or more of the points in thegroup that are displayed on list 1036. Once a point has been selected,the point can be added to one of the lists (1202 or 1206) with the AddSelected Points button 1212. The user tries to mark the same points onboth the anatomical display and the fluoro display. In the case of an EPprocedure, an effective technique is to identify the highest (mostsuperior) point in two or more of the pulmonary veins in each display.Once an appropriate number of corresponding points have been identifiedin both the anatomical and the x-ray views, the user can use the Adjustbutton 1024 to automatically find the best fit between the anatomicaland x-ray views. If the match is not satisfactory (i.e. if thecorresponding points from the x-ray view and the anatomical view are notsatisfactorily aligned), the user can reset the points and start overwith the Reset All button 1216. Alignment with the pane 1008 allowspoints and directions identified on the anatomical model to be displayedand used on the x-ray view, and vice versa. Thus the indicators incolumns 1028 and 1030 in pane 1032 can be checked to display fluoropoints, not just on the x-ray pane 1016, but also on the anatomical pane1014, and to display anatomical points, not just on the anatomical pane1016 but also on the x-ray pane 1016. As shown in FIG. 34A, the set offluoro points indicated by yellow circles in the anatomical pane 1014and the set of preoperative points indicated by red squares in theanatomical pane 1014 are fairly well aligned.

The automatic best fit alignment or registration can be implemented as acost function optimization procedure. Since preoperative image pointsare being registered to X-ray or fluoro-localized points, a naturalmeasure or cost function is the sum of the squared distances betweencorresponding points after a rigid transformation has been applied toone set of points, say the preoperative image points. The rigidtransformation in general consists of a rotation and a translation.Standard algorithms such as the so-called Procrustes method can be usedfor this purpose. In some cases the three dimensional image data may notrequire further orientational corrections, in which case only a best-fittranslation is required to implement the registration. In this lattercase it can be mathematically shown that a best-fit registration isachieved by matching the centroids of the 2 sets of points (thepreoperative image set and the fluoro-localized set); thus the distancevector between the corresponding centroids defines the requisitetranslation. Thus the “Adjust” button 1024 could implement any of theseschemes. Other intensity-based schemes familiar to those skilled in theart could also be used for the automatic registration process.

As shown in FIG. 49B, in the manual mode the pane 1008, has an adjustbox 1220 with an “All” pick button 1222, a “Fluoro” pick button 1224, a“Preop” pick button 1226. The pane 1008 also has a “Fluoro A” button1228 and a “Fluoro B” button 1230. The user can select the “Fluoro” pickbutton 1228, and then move the fluoro points displayed on thepreoperative anatomical image in pane 1014. The user can also select the“Preop” pick button 1224, and then move the preop points displayed onthe x-ray fluoroscopic image in pane 1016. Finally the user can selectthe “All” button, and then move the preop points displayed on the x-rayfluoroscopic image in pane 1016 and/or the fluoro points displayed onthe preoperative anatomical image in pane 1014. The user can use theFluoro A and Fluoro B buttons 1228 and 1230, which preferably displaythumbnail images, to switch the image that user uses to align with thepreoperative image points.

FIG. 34B is a view of the display 1000 similar to the view in FIG. 34A,except that the button 1070 has been pressed to display object 1058 cutin half by a plane in order to display the interior rather than theexterior of the object. The object 1058 has also been rotated slightlyin FIG. 34B from FIG. 34A. As shown in pane 1018 in FIG. 34B, the insideview is preferably taken along a vertical plane through the point ofrotation of the object 1058. The outside of the object 1058 is shown inFIG. 40, the inside of the object 1058 (rotated from FIG. 40) is shownin FIG. 41, and the standard user directions, several user selecteddirections, and the desired field direction are shown in the enlargedinside view of the object in FIG. 42.

FIG. 43A is another view of the display 1000 configured by the user withbutton 814, and selecting 2D Pane 1002, Navigations pane 1250, Presetspane 1006, and Bull's Eye pane 834. The user has also selected theobject navigation pane 1018, and the 30° LAO subpane from the 2D pane1002. In the object navigations pane, the user has selected (for exampleby right clicking on the representation 1058 of the object) to display arepresentation of the right atrium inside the representation 1058 of theobject.

The Navigations pane 1250 is adapted for storing and displaying defineddirections. As shown in FIG. 43A, the Navigations pane 1250 includes a“Navigations” list 1252 for listing various user defined directions. TheNavigations pane 1250 also includes a “Store” button 1254, an “Edit”button 1256, and a “Delete” button 1258. In this preferred embodiment,the Store button 1254 stores the current direction of the magneticfield, although when used with a non-magnetic navigation system it mightsimply store the current direction of the medical device. The Editbutton allows the user to edit the name of a previously stored point onthe Navigations list 1252. The Navigations pane 1250 also includes a“Reference” box 1260. The navigations box 1260 has two indicator windows1262 and 1264 for setting identifying information about a storeduser-defined direction. For example, the user can right click on thewindow to 1262 and 1264 to select color and shape, respectively, or thewindows 1262 and 1264 could be set up to receive text labels entered,for example with a keyboard. The reference box 1260 also includes a“Set” button 1266 to set the selected identifiers in the windows 1262and 1264 for a selected direction highlighted on the Navigations list1252. The reference box 1260 preferably also includes a “Remove” buttonto remove the identifiers for a selected direction highlighted on theNavigations list 1252.

The Bull's Eye pane 834 is adapted for manually or automaticallyadjusting the distal tip of the medical device.

FIG. 43B is a view similar to FIG. 43A, except that thesphere/hemisphere button has been operated in FIG. 43B to show theinterior of object 1058, and the object has been rotated.

FIG. 44 is another view of the display 1000 configured by the user withbutton 814, and selecting 2D Pane 1002, Navigations pane 1250, Presetspane 1006, and Bull's Eye pane 834. The user has also selected theobject navigation pane 1018, and the 30° LAO subpane from the 2D pane1002. In the object navigations pane 1018, the user has selected (forexample by right clicking on the representation 1058 of the object) todisplay a representation of the entire heart inside the representationof the object.

The Navigations pane 1250 is adapted for storing and displayinguser-defined directions. The Presets pane 1006 is adapted for recallingvarious preset directions and sets of directions. The Bull's Eye pane834 is adapted for manually pr automatically adjusting the distal tip ofthe medical device.

FIG. 45A is another view of the display 1000 configured by the user withbutton 814, and selecting 2D Pane 1002, Navigations pane 1250, Presetspane 1006, and Bull's Eye pane 834. The user has also selected theobject navigation pane 1018, and the 30° RAO subpane from the 2D pane1002. In the object navigations pane 1018, the user has selected (forexample by right clicking on the representation 1058 of the object) todisplay a representation of the left atrium inside the representation ofthe object.

The Navigations pane 1250 is adapted for storing and displaying defineddirections. The Presets pane 1006 is adapted for recalling variouspreset directions and sets of directions. The Bull's Eye pane 834 isadapted for manually or automatically adjusting the distal tip of themedical device.

FIG. 45B is a view similar to FIG. 45A, except that thesphere/hemisphere button has been operated in FIG. 45B to show theinterior of the representation 1058 of the object, and the object isshown rotated.

FIG. 46 is a view of a display 1300 from the CRT mode of the preferredembodiment of the user interface, selected by operating button 708 onthe menu 700 in FIG. 32. The display 1300 is similar to the display 1000and corresponding parts are identified with corresponding referencenumerals. The user has configured the display 1300 with button 814, andselecting 2D Pane 1002, Navigations pane 1252, Presets pane 1006, andBull's Eye pane 834. The user has also selected the object navigationpane 1018, and the 30° LAO subpane from the 2D pane 1002. In the objectnavigations pane 1018, the user has selected (for example by rightclicking on the representation 1058 of the object) to display arepresentation of the right atrium inside the representation 1058 of theobject.

FIG. 47 is a view of a display 1300 from the CRT mode of the preferredembodiment of the user interface, selected by operating button 708 onthe menu 700 in FIG. 32. The display 1300 is similar to the display 1000and corresponding parts are identified with corresponding referencenumerals. The user has configured the display 1300 with button 816, andselecting 2D Pane 1002, “Points & Constellations” pane 1004, Presetspane 1006, Bull's Eye pane 834, and Navigations pane 1250. The user hasalso selected the 3D pane 1012, the preoperative anatomical pane 1014(which in FIG. 47 shows the venous structure of the heart), and twofluoro panes 1066 to display the RAO and LAO images of the operatingregion.

As shown in FIG. 47, a vessel grouping is also possible in the Pointsand Constellations pane 1004. In the vessel grouping a series of pointsare joined as a continuous path, as a blood vessel. The user can usecolumns 1028 and 1030 to elect to display the vessel grouping on theimages in the fluoro panes 1066, and on the 3D pane 1012. As shown inFIG. 47, a vessel grouping 1400 is displayed in both fluoro panes 1066,and in the 3D pane 1012. The user can pick a direction parallel to thelongitudinal axis of the vessel grouping simply by selecting a point onthe vessel grouping. This is useful in navigating through vessels. Asshown in FIG. 47, the Bull's Eye pane 834 is in use so that the user canconveniently alter the indicated desired direction from the tangentialdirection to the vessel grouping.

FIG. 48A is a view of the display of an anatomical object navigationpane 1014. The user can identify points on the surface of the anatomicalobject 1126 by manipulating the object and pointing and clicking onpoints on its surface. These points can be grouped as constellations,for example splines, 1402,. 1404, 1404, and displayed on a 3D pane 1012.This facilitates visualization of the points selected directly fromsubject-derived anatomy.

While discussed above with respect to controlling a magnetic navigationsystem, it should be understood that any of the interfaces describedabove can be used to control any system for remotely orienting thedistal end of an elongate device, including but not limited to medicaldevices such as catheters and guide wires.

1. An interface for controlling a navigation system that orients the distal end of an elongate medical device in a selected direction in an operating region and moves the distal end of the medical device in the operating region, through the application of one or more control variables, the interface comprising: a control for applying a control variable to the navigation system corresponding to an angular orientation, a preselected amount greater than required to contact a surface, in a direction determined by the orientation of the medical device and a vector normal to the surface at the point of contact, to rotate the distal end of the elongate medical device in contact with a surface a selected amount
 2. The interface according to claim 1 wherein the control comprises a display for displaying the preselected amount, and buttons for increasing or decreasing the preselected amount.
 3. The interface according to claim 1 wherein the control comprises a slider for increasing or decreasing the preselected amount.
 4. A method for controlling a navigation system that orients the distal end of an elongate medical device in a selected direction in an operating region and moves the distal end of the medical device in the operating region, in response to at least one control variable, the method comprising: accepting a user input of an augmented angular orientation; determining the angular orientation of the distal end of the device to contact the surface at a selected point; determined an adjusted angular orientation from the determined angular orientation and the input augmented angular adjustment in a direction determined by the determined angular orientation and the surface normal vector at the selected point; and determining the control variables to cause the navigation system to orient the distal end of the medical device in the adjusted angular orientation.
 5. A method for controlling a navigation system that orients the distal end of an elongate medical device in a selected direction in an operating region and moves the distal end of the medical device in the operating region, in response to at least one control variable, the method comprising: applying control variables to cause the navigation system to orient the distal end of the elongate medical device in an orientation that is greater than necessary to contact the surface at a user selected point by a amount determined by a user input, in a direction determined by the surface normal vector at the selected point.
 6. An interface for controlling a navigation system that orients the distal end of an elongate medical device in a selected direction in an operating region and moves the distal end of the medical device in the operating region, through the application of one or more control variables, the interface comprising: a control for applying a control variable to the navigation system corresponding to a change in angular orientation of the device tip by a preselected amount greater than that required to barely contact a surface, in a direction determined by the orientation of the medical device and a vector normal to the surface at the point of contact.
 7. The interface according to claim 6 wherein the control comprises a display for displaying the preselected amount, and buttons for increasing or decreasing the preselected amount.
 8. The interface according to claim 6 wherein the control comprises a slider for increasing or decreasing the preselected amount.
 9. A method for controlling a navigation system that orients the distal end of an elongate medical device in a selected direction in an operating region and moves the distal end of the medical device in the operating region, in response to at least one control variable, the method comprising: accepting a user input of an augmented angular orientation; determining the angular orientation of the distal end of the device to contact the surface at a selected point; determining an adjusted angular orientation from the determined angular orientation and the input augmented angular adjustment in a direction determined by the determined angular orientation and the surface normal vector at the selected point; and determining the control variables to cause the navigation system to orient the distal end of the medical device in the adjusted angular orientation.
 10. A method for controlling a navigation system that orients the distal end of an elongate medical device in a selected direction in an operating region and moves the distal end of the medical device in the operating region, in response to at least one control variable, the method comprising: applying control variables to cause the navigation system to orient the distal end of the elongate medical device in an orientation that is greater than necessary to barely contact the surface at a user selected point by an amount determined by a user input, in a direction determined by the surface normal vector at the selected point.
 11. The interface of claim 6, where the control variable is a magnetic field.
 12. The method of claim 9 where the control variable is a magnetic field.
 13. The method of claim 10, where the control variable is a magnetic field. 